=Paper=
{{Paper
|id=Vol-2665/paper34
|storemode=property
|title=Landmine detection and minefield mapping with the help of multi-angle long-wave infrared hyperspectral data fused with the 3D terrain reconstruction
|pdfUrl=https://ceur-ws.org/Vol-2665/paper34.pdf
|volume=Vol-2665
|authors=Arsenii Golovin,Evgenii Sechak,Anatolii Demin
}}
==Landmine detection and minefield mapping with the help of multi-angle long-wave infrared hyperspectral data fused with the 3D terrain reconstruction ==
https://ceur-ws.org/Vol-2665/paper34.pdf
Landmine detection and minefield mapping with the
help of multi-angle long-wave infrared
hyperspectral data fused with the 3D terrain
reconstruction
Arsenii Golovin Evgenii Sechak Anatolii Demin
JSC «LOMO» Faculty of applied optics JSC «LOMO»
St. Petersburg, Russia ITMO University St. Petersburg, Russia
ibloogy@gmail.com St. Petersburg, Russia dav_60@mail.ru
evgenysechak@gmail.com
Abstract—The article proposes to use multi-angle II. DISTORTION IN HYPERSPECTRAL LANDMINE DETECTION
hyperspectral long-wave infrared remote sensing together with
three-dimensional reconstruction of the area to increase the Many modern methods for the detection of mines and
reliability of detection and reduce the frequency of false alarms minefields are based on the use of primary (external contour
when searching for subsurface objects - anti-personnel mines, and mine shape, contrast with relation to the surrounding
improvised explosive devices and unexploded ordnance in background, uniformity of the image within the mine
mountainous and hilly areas, where the use of minesweepers is contour, etc.) and secondary features (wilted vegetation,
difficult. Multi-angle remote sensing allows us to exclude the loosened soil, traces left by the mine-laying machine, etc.)
skipping of objects masked and laid at an angle and to separate [3,4].
the soil containing anomaly objects from ordinary soil and
surface irregularities. The concept of an optical-digital Hyperspectral imaging in the longwave infrared range
complex for minefield mapping is given, the main basis of (LWIR) allows detecting both primary and secondary
which is a hyperspectral device that receives data from two features. It has higher informativity before broadband
optical channels with divided them into the tens spectral infrared cameras since it allows dividing the spectral range of
channels in the longwave infrared range. One optic channel the infrared device into a group of dozens of different
scans the nadir and the second channel scans at an angle to the wavelengths with a high spectral resolution, the intensity of
soil surface. The complex also includes a camera of the visible each depends on the emissivity and allow to get the real
range, receiving a series of images in different spatial planes temperature of the soil surface [5,6,7].
for further three-dimensional reconstruction. A method for
obtaining and combining segmented hyperspectral data with a However, in mountainous and hilly terrain, hyperspectral
reconstructed digital terrain model is described for solving the imaging in nadir leads to inconsistent object detection
problems of detection of hidden ground and subsurface characteristics, since they are distorted due to local and
objects, reconnaissance, and planning of humanitarian global slopes of the terrain, which leads to a drop in the
demining missions on terrain with different slopes of relief. probability of detection, or an increase in false alarms.
Keywords—landmine detection, remote sensing, HSI, multi- Hyperspectral imaging, like other types of aerial
angle hyperspectral, LWIR, three-dimensional reconstruction, photography, is subject to several different distortions.
digital surface model. Changes in scale are usually associated with changes in
terrain height, but they may occur due to changes in flight
I. INTRODUCTION altitude along the route, for example, due to turbulence.
The danger of anti-personnel mines and unexploded Image skew distortions are due to roll and changes in the
ordnance is a serious problem. They pose a constant threat to pitch angle of the vehicle carrier due to maneuvering or wind
the lives and health of people, restrict the movement of gusts.
military forces, and deprive civilians of access to natural When scanning a nadir on the hilly and mountainous
resources. Currently, in more than sixty countries and terrain using the “push-broom” method, hyperspectral
regions of the world, millions of anti-personnel and anti- images are subject to relief and radial distortion, which leads
vehicle mines are pledged [1]. Every year, the number of to a sharp change in the objects buried at an angle to the
victims of anti-personnel mines is only growing. horizon on the crosslinked images along the flight path. An
In practice, the most effective way of clearing the area example of such a distortion for objects 1 and 3 is shown in
from mines during the fighting are special engineering Figure 1.
vehicles - minesweepers, and more mobile tractors with a Since, when detecting objects buried in the ground on the
mine trawl are used for humanitarian demining. However, LWIR hyperspectral images, as a rule, anomalous objects are
the use of such equipment is not always possible, for searched for with different thermal contrast with the
example, it is difficult for the mine trawl to overcome such surrounding soil, which does not always have obvious
obstacles as sections of the terrain with a sharp transition distinguishing features, as shown in the upper figure such
from descent to ascent and back, slopes, wide craters and objects during nadir scan can lead to various inconsistent
moats [2]. On this kind of terrain, the best way is to use detection characteristics — to the omission of objects, or an
remote methods from an unmanned aerial vehicle (UAV), increase in false alarms. Changes in the spectral emissivity
which can be used to search and map hidden subsurface between the thin layer of soil shown in the figure, and the
objects - anti-personnel mines, improvised explosive devices, environment may not be correctly interpreted by the anomaly
and unexploded ordnance, laid in the ground. detector and subsurface objects 1 and 3 will be skipped.
Copyright © 2020 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Image Processing and Earth Remote Sensing
more reliable since high accuracy of detection with a low
level of false alarms is required to solve problems associated
with demining.
III. MULTI-ANGLE LWIR HYPERSPECTRAL LANDMINE
DETECTION
To solve the above-mentioned problem, an approach is
proposed to implement remote search and mapping of the
hidden ground and subsurface objects on different slopes
using an optical-digital complex based on a multi-angle
LWIR hyperspectral scanning, together with a three-
dimensional reconstruction of the terrain, which allows
increasing the detection accuracy account of elimination of
relief distortions. Figure 3 shows an example of the
implementation of such a complex placed onboard a UAV in
the form of a multi-angle LWIR hyperspectral camera and a
high-resolution camera in the visible range.
Fig. 1. The case of hyperspectral imaging nadir scan on hilly terrain, when
there are various relief slopes along the flight path.
When processing the obtained hyperspectral data, there is Fig. 3. Optical-digital complex located onboard the UAV.
also the problem of spectral distortions of the same objects
located on different global and local slopes. Figure 2 shows The hyperspectral camera receives data in 2 separate
an example of the non-linear distortion of the spectral spatial channels, one of which collects reflected radiation at a
characteristics of woody vegetation that were obtained in nadir scan, and the other has an angular displacement of 45°
areas with slopes from 0° to 45° [8]. relative to the nadir axis. This makes it possible to obtain
hyperspectral images in an oblique projection (military
Errors in the detection of anomalous objects on perspective), which makes it possible to search for ground
hyperspectral data due to changes in scale, radial and subsurface objects on various local and global
displacement, obliquity, relief, and spectral distortions lead inhomogeneities of the relief [9]. An example of the
to false alarms and the omission of small engineering objects transverse scanning by the second channel of the
(unexploded ordnance, mines, and improvised explosive hyperspectral camera of the terrain indicated earlier is shown
devices) installed on the soil surface and laid in the ground. in Figure 4, the large arrow indicates the direction of flight.
Also, these errors reduce the reliability of the data and make The previously distorted objects 1 and 3 on the hyperspectral
it difficult to transfer detected objects to maps in GIS for frames corresponding to the scanning direction can be easily
solving problems related to humanitarian demining, distinguished by anomaly detectors.
reconnaissance, planning the movement of people,
equipment, etc.
Fig. 4. An example of the transverse scanning by the second channel of
the hyperspectral camera, when there are various relief slopes along the
flight path.
When detecting small engineering objects in
mountainous and hilly parts of the area, scanning with a
Fig. 2. Woody vegetation spectra distorted by terrain topography. multi-angle LWIR hyperspectral camera should be
conducted in the transverse and longitudinal directions with
This problem can be minimized by flying at high overlapping more than 50%, as shown in Figure 5. The blue
altitudes, however, the process of detecting small dot is the scan start point, the green dot is the end of
engineering objects is usually carried out at relatively low scanning. The black line is the flight path of the UAV, the
altitudes (50-200 m), due to the low resolution of LWIR areas with shading in the right and left direction are the
sensors. The application of topographic radiometric sensing bands for different directions of data collection with
correction algorithms to hyperspectral data can improve the a multi-angle LWIR hyperspectral camera. The area shaded
final image, but the physics of data collection in one by the grid is the overlapping area of the hyperspectral data
projection (in nadir) does not make the detection process for different scanning directions.
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 150
Image Processing and Earth Remote Sensing
expressed as the sum of the background matrix B and the
matrix of anomalies S. The background matrix B represents
the spectral vectors of the main ground objects throughout
the image scene and it is assumed that it lies on a low-
dimensional subspace with low-level properties. In the
matrix of anomalies S, collected spectral vectors of small and
Fig. 5. Data collection by optical-digital complex in the longitudinal and
unlikely ground objects, collected in columns C. The
transverse directions. corresponding columns C of the background matrix are zero.
Matrices Y, B, and S related as:
As a result of the remote sensing of the territory with a
multi-angle hyperspectral camera, 8 hyperspectral data cubes
are assembled in the transverse and longitudinal directions: 4
in the longitudinal EW (east-west) and WE (west-east) for where bi and si is the ith column of matrices B and S,
two channels and 4 in the transverse SN (south-north) and accordingly, r is the dimension of the subspace of the matrix
NS (north-south). When the hyperspectral data is stitched, B, and U is the basis of the column of the subspace of the
with a link to high-resolution photographs of high resolution matrix B. Limitation Rank(B) = r – ensures low subspace B.
in the visible range, radiometric correction and geo A limitation (I-U(UTU)-1UT)si0, i C is that the
correction are carried out. The next step is to use the method anomalies do not lie in the columns of the subspace. To
of detecting anomalous objects, based on a modification of speed up the processing of four hyperspectral data cubes
the principal component method and providing good obtained in the nadir scan, we propose to use the background
performance with low computational complexity due to the matrix obtained during the processing of the first hypercube
use of random selection and prediction [10]. to calculate the matrix of anomalies on the three remaining
Hyperspectral data are considered as a set of band hypercube data. The resulting 4 maps of anomalies in nadir
vectors: scans are further combined into one general map to increase
the reliability of the detection of anomalous objects and
reduce the frequency of false detections. The anomaly
where M - is the number of bands, and N - is the number of detector is also used for hyperspectral data obtained by the
pixels. The background matrix and the anomaly matrix are second channel of the hyperspectral camera. As a result, we
denoted as B HMN and S HMN. Anomaly detection get 5 different anomaly maps. The general scheme of data
consists of separating the anomaly from the background, so processing by an optical-digital complex is shown in Figure
the matrix of the corrected hyperspectral image Y can be 6.
Fig. 6. Data collection by optical-digital complex in the longitudinal and transverse directions.
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 151
Image Processing and Earth Remote Sensing
In turn, a cloud of points is built from high-resolution hilly or mountainous areas, where the use of minesweepers
images obtained by a camera of the visible range from is difficult.
different angles using three-dimensional reconstruction
algorithms of a digital terrain model. These point clouds REFERENCES
are classified into several areas corresponding to local and [1] Why Landmines Are Still a Problem - International Campaign to
global slopes using the spatial classification method, which Ban Landmines (ICBL) [Online]. URL: http://www.icbl.org/en-
determines the class of a group of points based on its gb/problem/why-landmines-are-still-a-problem.aspx (05.10.2019).
spatial position, shape, and size. Medium shear clustering [2] A. Khamis, “Minesweepers: Towards a Landmine-free World,”
IEEE Canada International Humanitarian Technology Conference
is used to perform spatial segmentation. This method is a (IHTC), 2015.
non-parametric clustering technique that does not require [3] N.T. Thanh, H. Sahli and D.N. Hao, “Infrared Thermography for
knowledge of the number of clusters or the shape of these Buried Landmine Detection: Inverse Problem Setting,” IEEE
clusters [11,12]. The mean shift algorithm groups the point Transactions on Geo-science and Remote Sensing, vol. 46, no. 12,
cloud into separate clusters, shifting the mean value to a pp. 3987-4004, 2008.
denser direction, where most of the points are located. [4] P.A. Jacobs, “Thermal Infrared Characterization of Ground Targets
These clusters are fed to the dispersion-based segmentation and Backgrounds,” SPIE Tutorial Texts in Optical Engineering,
vol. TT70, p. 190, 2006.
method, which in turn further segments the previously
obtained clusters [13-17]. [5] E.M. Winter, M.A. Miller, C.G. Simi, A.B. Hill, T.J. Williams, D.
Hampton, M. Wood and J. Zadnick, “Mine detection experiments
Segmented point cloud data and multi-angle anomaly using hyperspectral sensors,” Detection and Remediation
Technologies for Mines and Minelike Targets IX, Proc. SPIE, vol.
maps are fed into the neural network [14], where, at the 5415, 2004.
training stage, the corresponding pixels from the multi- [6] J.E. McFee, C. Anger, S. Achal and T. Ivanco, “Landmine
angle anomaly maps are assigned to each class of local and detection using passive hyperspectral imaging,” Chemical and
global slope. After the error in each neuron is less than the Biological Sensing VIII, vol. 6554, 2007.
predetermined tolerance, the neural network is considered [7] M. Gagnon, P. Lagueux, J.P. Gagnon, S. Savary, P. Tremblay, V.
to be trained. In the end, the work of the neural network Farley, E. Guyot and M. Chamberland, “Airborne thermal infrared
creates a three-dimensional map of the anomalies, which hyperspectral imaging of buried objects,” Electro-Optical Remote
Sensing, Photonic Technologies and Applications IX, vol. 9649,
are compared with the terrain. From the data obtained at 2015.
the early stages of processing, a three-dimensional model
[8] A.N. Grigoriev, “Topographic radiometric correction of space
of the surface of the visible range is constructed, which in hyperspectral imagery data,” Abstracts of the reports of the
turn is combined with the obtained three-dimensional map scientific and technical conference "Hyperspectral devices and
of anomalies, which are highlighted for the operator with technologies", Krasnogorsk, Russia, pp. 130-132, 2013.
bright color. [9] A.D. Golovin and A.V. Demin, “Optical-digital complex for
detection of remote mines and mapping of minefields,” Journal of
IV. CONCLUSION Physics: Conference Series, vol. 1096, no. 1, pp. 012-011, 2018.
[10] T. Zhou and D. Godec, “Randomized low-rank & sparse matrix
An approach to the implementation of remote search decomposition in noisy case,” Proceedings of the International
and mapping of hidden land and subsurface objects - anti- conference on machine learning, USA, Bellevue, WA, vol. 46, no.
personnel mines, improvised explosive devices, and 12, pp. 33-40, 2011.
unexploded ordnance, buried into the ground at a shallow [11] H. Woo, E. Kang, S. Wang and K.H. Lee, “A new segmentation
depth, on the territory with different slopes using an method for point cloud data,” International Journal of Machine
optical-digital complex based on multi-view long-wave Tools and Manufacture, vol. 42, no. 2, pp. 167-178, 2002.
infrared hyperspectral scanning together with a three- [12] M. Fashing and C. Tomasi, “Mean shift is a bound optimization,”
IEEE Transactions on Pattern Analysis and Machine Intelligence,
dimensional terrain reconstruction is proposed. The vol. 27, no. 3, pp. 471-474, 2005.
method allows for increasing the reliability of detection by
[13] D. Comaniciu and P. Meer, “Mean shift: a robust approach toward
eliminating the relief and spectral distortion. The feature space analysis,” IEEE Transactions on Pattern Analysis and
application of the anomaly detection method in multi-view Machine Intelligence, vol. 24, no. 5, pp. 603-619, 2002.
hyperspectral images based on the decomposition of the [14] A.V. Mingalev, A.V. Belov, I.M. Gabdullin, R.R. Agafonova and
hyperspectral image matrix into the background, including S.N. Shusharin, “Test-object recognition in thermal images,”
spectral vectors of main ground objects in the image scene Computer Optics, vol. 43, no. 3, pp. 402-411, 2019.
and an anomaly matrix containing spectral vectors of small [15] B. Xiang, J. Yao and X. Lu, “Segmentation-based classification for
and unlikely ground objects, provides good performance 3D point clouds in the road environment,” International Journal of
Remote Sensing, vol. 39, no. 19, 2018.
with low computational complexity. A method of
[16] L.I. Perlovsky and M.M. McManus, “Maximum likelihood neural
combining local slope clusters classified from points networks for sensor fusion and adaptive classification,” Neural
clouds of terrain with a segment of anomaly map of one of Networks, vol. 4, no. 1, pp. 89-102, 1991.
the hyperspectral images taken from different angles is [17] Yu.V. Vizilter, V.S. Gorbatsevich and S.Y. Zheltov, “Structure-
proposed. The technique allows excluding to a minimum functional analysis and synthesis of deep convolutional neural
both spectral and spatial topographic distortions inherent in networks,” Computer Optics, vol. 43, no. 5, pp. 886-900, 2019.
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 152