Landmine detection and minefield mapping with the help of multi-angle long-wave infrared hyperspectral data fused with the 3D terrain reconstruction

Abstract

-The article proposes to use multi-angle hyperspectral long-wave infrared remote sensing together with three-dimensional reconstruction of the area to increase the reliability of detection and reduce the frequency of false alarms when searching for subsurface objects - anti-personnel mines, improvised explosive devices and unexploded ordnance in mountainous and hilly areas, where the use of minesweepers is difficult. Multi-angle remote sensing allows us to exclude the skipping of objects masked and laid at an angle and to separate the soil containing anomaly objects from ordinary soil and surface irregularities. The concept of an optical-digital complex for minefield mapping is given, the main basis of which is a hyperspectral device that receives data from two optical channels with divided them into the tens spectral channels in the longwave infrared range. One optic channel scans the nadir and the second channel scans at an angle to the soil surface. The complex also includes a camera of the visible range, receiving a series of images in different spatial planes for further three-dimensional reconstruction. A method for obtaining and combining segmented hyperspectral data with a reconstructed digital terrain model is described for solving the problems of detection of hidden ground and subsurface objects, reconnaissance, and planning of humanitarian demining missions on terrain with different slopes of relief.

Keywords

• landmine detection
• remote sensing
• HSI
• multiangle hyperspectral
• LWIR
• three-dimensional reconstruction
• digital surface model

I. INTRODUCTION

The danger of anti-personnel mines and unexploded ordnance is a serious problem. They pose a constant threat to the lives and health of people, restrict the movement of military forces, and deprive civilians of access to natural resources. Currently, in more than sixty countries and regions of the world, millions of anti-personnel and antivehicle mines are pledged [1]. Every year, the number of victims of anti-personnel mines is only growing.

In practice, the most effective way of clearing the area from mines during the fighting are special engineering vehicles - minesweepers, and more mobile tractors with a mine trawl are used for humanitarian demining. However, the use of such equipment is not always possible, for example, it is difficult for the mine trawl to overcome such obstacles as sections of the terrain with a sharp transition from descent to ascent and back, slopes, wide craters and moats [ 2 ]. On this kind of terrain, the best way is to use remote methods from an unmanned aerial vehicle (UAV), which can be used to search and map hidden subsurface objects - anti-personnel mines, improvised explosive devices, and unexploded ordnance, laid in the ground.

II. DISTORTION IN HYPERSPECTRAL LANDMINE DETECTION

Many modern methods for the detection of mines and minefields are based on the use of primary (external contour and mine shape, contrast with relation to the surrounding background, uniformity of the image within the mine contour, etc.) and secondary features (wilted vegetation, loosened soil, traces left by the mine-laying machine, etc.) [ 3,4 ].

Hyperspectral imaging in the longwave infrared range (LWIR) allows detecting both primary and secondary features. It has higher informativity before broadband infrared cameras since it allows dividing the spectral range of the infrared device into a group of dozens of different wavelengths with a high spectral resolution, the intensity of each depends on the emissivity and allow to get the real temperature of the soil surface [ 5,6,7 ].

However, in mountainous and hilly terrain, hyperspectral imaging in nadir leads to inconsistent object detection characteristics, since they are distorted due to local and global slopes of the terrain, which leads to a drop in the probability of detection, or an increase in false alarms.

Hyperspectral imaging, like other types of aerial photography, is subject to several different distortions. Changes in scale are usually associated with changes in terrain height, but they may occur due to changes in flight altitude along the route, for example, due to turbulence. Image skew distortions are due to roll and changes in the pitch angle of the vehicle carrier due to maneuvering or wind gusts.

When scanning a nadir on the hilly and mountainous terrain using the “push-broom” method, hyperspectral images are subject to relief and radial distortion, which leads to a sharp change in the objects buried at an angle to the horizon on the crosslinked images along the flight path. An example of such a distortion for objects 1 and 3 is shown in Figure 1.

Since, when detecting objects buried in the ground on the LWIR hyperspectral images, as a rule, anomalous objects are searched for with different thermal contrast with the surrounding soil, which does not always have obvious distinguishing features, as shown in the upper figure such objects during nadir scan can lead to various inconsistent detection characteristics — to the omission of objects, or an increase in false alarms. Changes in the spectral emissivity between the thin layer of soil shown in the figure, and the environment may not be correctly interpreted by the anomaly detector and subsurface objects 1 and 3 will be skipped.

When processing the obtained hyperspectral data, there is also the problem of spectral distortions of the same objects located on different global and local slopes. Figure 2 shows an example of the non-linear distortion of the spectral characteristics of woody vegetation that were obtained in areas with slopes from 0° to 45° [ 8 ].

Errors in the detection of anomalous objects on hyperspectral data due to changes in scale, radial displacement, obliquity, relief, and spectral distortions lead to false alarms and the omission of small engineering objects (unexploded ordnance, mines, and improvised explosive devices) installed on the soil surface and laid in the ground. Also, these errors reduce the reliability of the data and make it difficult to transfer detected objects to maps in GIS for solving problems related to humanitarian demining, reconnaissance, planning the movement of people, equipment, etc.

This problem can be minimized by flying at high altitudes, however, the process of detecting small engineering objects is usually carried out at relatively low altitudes (50-200 m), due to the low resolution of LWIR sensors. The application of topographic radiometric correction algorithms to hyperspectral data can improve the final image, but the physics of data collection in one projection (in nadir) does not make the detection process more reliable since high accuracy of detection with a low level of false alarms is required to solve problems associated with demining.

III. MULTI-ANGLE LWIR HYPERSPECTRAL LANDMINE

To solve the above-mentioned problem, an approach is proposed to implement remote search and mapping of the hidden ground and subsurface objects on different slopes using an optical-digital complex based on a multi-angle LWIR hyperspectral scanning, together with a threedimensional reconstruction of the terrain, which allows increasing the detection accuracy account of elimination of relief distortions. Figure 3 shows an example of the implementation of such a complex placed onboard a UAV in the form of a multi-angle LWIR hyperspectral camera and a high-resolution camera in the visible range.

The hyperspectral camera receives data in 2 separate spatial channels, one of which collects reflected radiation at a nadir scan, and the other has an angular displacement of 45° relative to the nadir axis. This makes it possible to obtain hyperspectral images in an oblique projection (military perspective), which makes it possible to search for ground and subsurface objects on various local and global inhomogeneities of the relief [ 9 ]. An example of the transverse scanning by the second channel of the hyperspectral camera of the terrain indicated earlier is shown in Figure 4, the large arrow indicates the direction of flight. The previously distorted objects 1 and 3 on the hyperspectral frames corresponding to the scanning direction can be easily distinguished by anomaly detectors.

When detecting small engineering objects in mountainous and hilly parts of the area, scanning with a multi-angle LWIR hyperspectral camera should be conducted in the transverse and longitudinal directions with overlapping more than 50%, as shown in Figure 5. The blue dot is the scan start point, the green dot is the end of scanning. The black line is the flight path of the UAV, the areas with shading in the right and left direction are the sensing bands for different directions of data collection with a multi-angle LWIR hyperspectral camera. The area shaded by the grid is the overlapping area of the hyperspectral data for different scanning directions.

As a result of the remote sensing of the territory with a multi-angle hyperspectral camera, 8 hyperspectral data cubes are assembled in the transverse and longitudinal directions: 4 in the longitudinal EW (east-west) and WE (west-east) for two channels and 4 in the transverse SN (south-north) and NS (north-south). When the hyperspectral data is stitched, with a link to high-resolution photographs of high resolution in the visible range, radiometric correction and geo correction are carried out. The next step is to use the method of detecting anomalous objects, based on a modification of the principal component method and providing good performance with low computational complexity due to the use of random selection and prediction [ 10 ].

Hyperspectral data are considered as a set of band vectors: where M - is the number of bands, and N - is the number of pixels. The background matrix and the anomaly matrix are denoted as B  HMN and S  HMN. Anomaly detection consists of separating the anomaly from the background, so the matrix of the corrected hyperspectral image Y can be expressed as the sum of the background matrix B and the matrix of anomalies S. The background matrix B represents the spectral vectors of the main ground objects throughout the image scene and it is assumed that it lies on a lowdimensional subspace with low-level properties. In the matrix of anomalies S, collected spectral vectors of small and unlikely ground objects, collected in columns C. The corresponding columns C of the background matrix are zero. Matrices Y, B, and S related as: where bi and si is the ith column of matrices B and S, accordingly, r is the dimension of the subspace of the matrix B, and U is the basis of the column of the subspace of the matrix B. Limitation Rank(B) = r – ensures low subspace B. A limitation (I-U(UTU)-1UT)si0, i  C is that the anomalies do not lie in the columns of the subspace. To speed up the processing of four hyperspectral data cubes obtained in the nadir scan, we propose to use the background matrix obtained during the processing of the first hypercube to calculate the matrix of anomalies on the three remaining hypercube data. The resulting 4 maps of anomalies in nadir scans are further combined into one general map to increase the reliability of the detection of anomalous objects and reduce the frequency of false detections. The anomaly detector is also used for hyperspectral data obtained by the second channel of the hyperspectral camera. As a result, we get 5 different anomaly maps. The general scheme of data processing by an optical-digital complex is shown in Figure 6.

In turn, a cloud of points is built from high-resolution images obtained by a camera of the visible range from different angles using three-dimensional reconstruction algorithms of a digital terrain model. These point clouds are classified into several areas corresponding to local and global slopes using the spatial classification method, which determines the class of a group of points based on its spatial position, shape, and size. Medium shear clustering is used to perform spatial segmentation. This method is a non-parametric clustering technique that does not require knowledge of the number of clusters or the shape of these clusters [ 11,12 ]. The mean shift algorithm groups the point cloud into separate clusters, shifting the mean value to a denser direction, where most of the points are located. These clusters are fed to the dispersion-based segmentation method, which in turn further segments the previously obtained clusters [ 13-17 ].

Segmented point cloud data and multi-angle anomaly maps are fed into the neural network [ 14 ], where, at the training stage, the corresponding pixels from the multiangle anomaly maps are assigned to each class of local and global slope. After the error in each neuron is less than the predetermined tolerance, the neural network is considered to be trained. In the end, the work of the neural network creates a three-dimensional map of the anomalies, which are compared with the terrain. From the data obtained at the early stages of processing, a three-dimensional model of the surface of the visible range is constructed, which in turn is combined with the obtained three-dimensional map of anomalies, which are highlighted for the operator with bright color.

IV. CONCLUSION

An approach to the implementation of remote search and mapping of hidden land and subsurface objects - antipersonnel mines, improvised explosive devices, and unexploded ordnance, buried into the ground at a shallow depth, on the territory with different slopes using an optical-digital complex based on multi-view long-wave infrared hyperspectral scanning together with a threedimensional terrain reconstruction is proposed. The method allows for increasing the reliability of detection by eliminating the relief and spectral distortion. The application of the anomaly detection method in multi-view hyperspectral images based on the decomposition of the hyperspectral image matrix into the background, including spectral vectors of main ground objects in the image scene and an anomaly matrix containing spectral vectors of small and unlikely ground objects, provides good performance with low computational complexity. A method of combining local slope clusters classified from points clouds of terrain with a segment of anomaly map of one of the hyperspectral images taken from different angles is proposed. The technique allows excluding to a minimum both spectral and spatial topographic distortions inherent in hilly or mountainous areas, where the use of minesweepers is difficult.

References

Why Landmines Are Still a Problem - International Campaign to Ban Landmines (ICBL) [Online] . URL: http://www.icbl.org/engb/problem/why -landmines-are-still-a-problem . aspx (05.10 . 2019 ).

[2] A. Khamis , “Minesweepers: Towards a Landmine- free World, ” IEEE Canada International Humanitarian Technology Conference (IHTC) , 2015 .

[3] N.T. Thanh , H. Sahli and D.N. Hao , “ Infrared Thermography for Buried Landmine Detection: Inverse Problem Setting,” IEEE Transactions on Geo-science and Remote Sensing , vol. 46 , no. 12 , pp. 3987 - 4004 , 2008 .

[4] P.A. Jacobs , “ Thermal Infrared Characterization of Ground Targets and Backgrounds,” SPIE Tutorial Texts in Optical Engineering, vol. TT70 , p. 190 , 2006 .

[5] E.M. Winter , M.A. Miller , C.G. Simi , A.B. Hill , T.J. Williams , D. Hampton , M. Wood and J. Zadnick , “ Mine detection experiments using hyperspectral sensors,” Detection and Remediation Technologies for Mines and Minelike Targets IX , Proc. SPIE , vol. 5415 , 2004 .

[6] J.E. McFee , C. Anger , S. Achal and T. Ivanco , “ Landmine detection using passive hyperspectral imaging,” Chemical and Biological Sensing VIII , vol. 6554 , 2007 .

Farley , E. Guyot and M. Chamberland , “ Airborne thermal infrared hyperspectral imaging of buried objects,” Electro-Optical Remote Sensing, Photonic Technologies and Applications IX , vol. 9649 , 2015 .

[8] A.N. Grigoriev , “ Topographic radiometric correction of space hyperspectral imagery data,” Abstracts of the reports of the scientific and technical conference "Hyperspectral devices and technologies" , Krasnogorsk, Russia, pp. 130 - 132 , 2013 .

[9] A.D. Golovin and A.V. Demin , “ Optical-digital complex for detection of remote mines and mapping of minefields , ” Journal of Physics: Conference Series , vol. 1096 , no. 1 , pp. 012 - 011 , 2018 .

[10] T. Zhou and D. Godec , “ Randomized low-rank & sparse matrix decomposition in noisy case , ” Proceedings of the International conference on machine learning , USA, Bellevue, WA, vol. 46 , no. 12 , pp. 33 - 40 , 2011 .

[11] H. Woo , E. Kang , S. Wang and K.H. Lee , “ A new segmentation method for point cloud data ,” International Journal of Machine Tools and Manufacture , vol. 42 , no. 2 , pp. 167 - 178 , 2002 .

[12] M. Fashing and C. Tomasi , “ Mean shift is a bound optimization , ” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 27 , no. 3 , pp. 471 - 474 , 2005 .

[13] D. Comaniciu and P. Meer , “ Mean shift: a robust approach toward feature space analysis , ” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 24 , no. 5 , pp. 603 - 619 , 2002 .

[14] A.V. Mingalev , A.V. Belov , I.M. Gabdullin , R.R. Agafonova and S.N. Shusharin , “ Test-object recognition in thermal images ,” Computer Optics , vol. 43 , no. 3 , pp. 402 - 411 , 2019 .

[15] B. Xiang , J. Yao and X. Lu , “ Segmentation-based classification for 3D point clouds in the road environment ,” International Journal of Remote Sensing , vol. 39 , no. 19 , 2018 .

[16] L.I. Perlovsky and M.M. McManus , “ Maximum likelihood neural networks for sensor fusion and adaptive classification,” Neural Networks , vol. 4 , no. 1 , pp. 89 - 102 , 1991 .

[17] Yu . V. Vizilter , V.S. Gorbatsevich and S.Y. Zheltov , “ Structurefunctional analysis and synthesis of deep convolutional neural networks , ” Computer Optics , vol. 43 , no. 5 , pp. 886 - 900 , 2019 .