Landmine detection and minefield mapping with the help of multi-angle long-wave infrared hyperspectral data fused with the 3D terrain reconstruction Arsenii Golovin Evgenii Sechak Anatolii Demin JSC «LOMO» Faculty of applied optics JSC «LOMO» St. Petersburg, Russia ITMO University St. Petersburg, Russia ibloogy@gmail.com St. Petersburg, Russia dav_60@mail.ru evgenysechak@gmail.com Abstract—The article proposes to use multi-angle II. DISTORTION IN HYPERSPECTRAL LANDMINE DETECTION hyperspectral long-wave infrared remote sensing together with three-dimensional reconstruction of the area to increase the Many modern methods for the detection of mines and reliability of detection and reduce the frequency of false alarms minefields are based on the use of primary (external contour when searching for subsurface objects - anti-personnel mines, and mine shape, contrast with relation to the surrounding improvised explosive devices and unexploded ordnance in background, uniformity of the image within the mine mountainous and hilly areas, where the use of minesweepers is contour, etc.) and secondary features (wilted vegetation, difficult. Multi-angle remote sensing allows us to exclude the loosened soil, traces left by the mine-laying machine, etc.) skipping of objects masked and laid at an angle and to separate [3,4]. the soil containing anomaly objects from ordinary soil and surface irregularities. The concept of an optical-digital Hyperspectral imaging in the longwave infrared range complex for minefield mapping is given, the main basis of (LWIR) allows detecting both primary and secondary which is a hyperspectral device that receives data from two features. It has higher informativity before broadband optical channels with divided them into the tens spectral infrared cameras since it allows dividing the spectral range of channels in the longwave infrared range. One optic channel the infrared device into a group of dozens of different scans the nadir and the second channel scans at an angle to the wavelengths with a high spectral resolution, the intensity of soil surface. The complex also includes a camera of the visible each depends on the emissivity and allow to get the real range, receiving a series of images in different spatial planes temperature of the soil surface [5,6,7]. for further three-dimensional reconstruction. A method for obtaining and combining segmented hyperspectral data with a However, in mountainous and hilly terrain, hyperspectral reconstructed digital terrain model is described for solving the imaging in nadir leads to inconsistent object detection problems of detection of hidden ground and subsurface characteristics, since they are distorted due to local and objects, reconnaissance, and planning of humanitarian global slopes of the terrain, which leads to a drop in the demining missions on terrain with different slopes of relief. probability of detection, or an increase in false alarms. Keywords—landmine detection, remote sensing, HSI, multi- Hyperspectral imaging, like other types of aerial angle hyperspectral, LWIR, three-dimensional reconstruction, photography, is subject to several different distortions. digital surface model. Changes in scale are usually associated with changes in terrain height, but they may occur due to changes in flight I. INTRODUCTION altitude along the route, for example, due to turbulence. The danger of anti-personnel mines and unexploded Image skew distortions are due to roll and changes in the ordnance is a serious problem. They pose a constant threat to pitch angle of the vehicle carrier due to maneuvering or wind the lives and health of people, restrict the movement of gusts. military forces, and deprive civilians of access to natural When scanning a nadir on the hilly and mountainous resources. Currently, in more than sixty countries and terrain using the “push-broom” method, hyperspectral regions of the world, millions of anti-personnel and anti- images are subject to relief and radial distortion, which leads vehicle mines are pledged [1]. Every year, the number of to a sharp change in the objects buried at an angle to the victims of anti-personnel mines is only growing. horizon on the crosslinked images along the flight path. An In practice, the most effective way of clearing the area example of such a distortion for objects 1 and 3 is shown in from mines during the fighting are special engineering Figure 1. vehicles - minesweepers, and more mobile tractors with a Since, when detecting objects buried in the ground on the mine trawl are used for humanitarian demining. However, LWIR hyperspectral images, as a rule, anomalous objects are the use of such equipment is not always possible, for searched for with different thermal contrast with the example, it is difficult for the mine trawl to overcome such surrounding soil, which does not always have obvious obstacles as sections of the terrain with a sharp transition distinguishing features, as shown in the upper figure such from descent to ascent and back, slopes, wide craters and objects during nadir scan can lead to various inconsistent moats [2]. On this kind of terrain, the best way is to use detection characteristics — to the omission of objects, or an remote methods from an unmanned aerial vehicle (UAV), increase in false alarms. Changes in the spectral emissivity which can be used to search and map hidden subsurface between the thin layer of soil shown in the figure, and the objects - anti-personnel mines, improvised explosive devices, environment may not be correctly interpreted by the anomaly and unexploded ordnance, laid in the ground. detector and subsurface objects 1 and 3 will be skipped. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) Image Processing and Earth Remote Sensing more reliable since high accuracy of detection with a low level of false alarms is required to solve problems associated with demining. III. MULTI-ANGLE LWIR HYPERSPECTRAL LANDMINE DETECTION To solve the above-mentioned problem, an approach is proposed to implement remote search and mapping of the hidden ground and subsurface objects on different slopes using an optical-digital complex based on a multi-angle LWIR hyperspectral scanning, together with a three- dimensional reconstruction of the terrain, which allows increasing the detection accuracy account of elimination of relief distortions. Figure 3 shows an example of the implementation of such a complex placed onboard a UAV in the form of a multi-angle LWIR hyperspectral camera and a high-resolution camera in the visible range. Fig. 1. The case of hyperspectral imaging nadir scan on hilly terrain, when there are various relief slopes along the flight path. When processing the obtained hyperspectral data, there is Fig. 3. Optical-digital complex located onboard the UAV. also the problem of spectral distortions of the same objects located on different global and local slopes. Figure 2 shows The hyperspectral camera receives data in 2 separate an example of the non-linear distortion of the spectral spatial channels, one of which collects reflected radiation at a characteristics of woody vegetation that were obtained in nadir scan, and the other has an angular displacement of 45° areas with slopes from 0° to 45° [8]. relative to the nadir axis. This makes it possible to obtain hyperspectral images in an oblique projection (military Errors in the detection of anomalous objects on perspective), which makes it possible to search for ground hyperspectral data due to changes in scale, radial and subsurface objects on various local and global displacement, obliquity, relief, and spectral distortions lead inhomogeneities of the relief [9]. An example of the to false alarms and the omission of small engineering objects transverse scanning by the second channel of the (unexploded ordnance, mines, and improvised explosive hyperspectral camera of the terrain indicated earlier is shown devices) installed on the soil surface and laid in the ground. in Figure 4, the large arrow indicates the direction of flight. Also, these errors reduce the reliability of the data and make The previously distorted objects 1 and 3 on the hyperspectral it difficult to transfer detected objects to maps in GIS for frames corresponding to the scanning direction can be easily solving problems related to humanitarian demining, distinguished by anomaly detectors. reconnaissance, planning the movement of people, equipment, etc. Fig. 4. An example of the transverse scanning by the second channel of the hyperspectral camera, when there are various relief slopes along the flight path. When detecting small engineering objects in mountainous and hilly parts of the area, scanning with a Fig. 2. Woody vegetation spectra distorted by terrain topography. multi-angle LWIR hyperspectral camera should be conducted in the transverse and longitudinal directions with This problem can be minimized by flying at high overlapping more than 50%, as shown in Figure 5. The blue altitudes, however, the process of detecting small dot is the scan start point, the green dot is the end of engineering objects is usually carried out at relatively low scanning. The black line is the flight path of the UAV, the altitudes (50-200 m), due to the low resolution of LWIR areas with shading in the right and left direction are the sensors. The application of topographic radiometric sensing bands for different directions of data collection with correction algorithms to hyperspectral data can improve the a multi-angle LWIR hyperspectral camera. The area shaded final image, but the physics of data collection in one by the grid is the overlapping area of the hyperspectral data projection (in nadir) does not make the detection process for different scanning directions. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 150 Image Processing and Earth Remote Sensing expressed as the sum of the background matrix B and the matrix of anomalies S. The background matrix B represents the spectral vectors of the main ground objects throughout the image scene and it is assumed that it lies on a low- dimensional subspace with low-level properties. In the matrix of anomalies S, collected spectral vectors of small and Fig. 5. Data collection by optical-digital complex in the longitudinal and unlikely ground objects, collected in columns C. The transverse directions. corresponding columns C of the background matrix are zero. Matrices Y, B, and S related as: As a result of the remote sensing of the territory with a multi-angle hyperspectral camera, 8 hyperspectral data cubes are assembled in the transverse and longitudinal directions: 4 in the longitudinal EW (east-west) and WE (west-east) for where bi and si is the ith column of matrices B and S, two channels and 4 in the transverse SN (south-north) and accordingly, r is the dimension of the subspace of the matrix NS (north-south). When the hyperspectral data is stitched, B, and U is the basis of the column of the subspace of the with a link to high-resolution photographs of high resolution matrix B. Limitation Rank(B) = r – ensures low subspace B. in the visible range, radiometric correction and geo A limitation (I-U(UTU)-1UT)si0, i  C is that the correction are carried out. The next step is to use the method anomalies do not lie in the columns of the subspace. To of detecting anomalous objects, based on a modification of speed up the processing of four hyperspectral data cubes the principal component method and providing good obtained in the nadir scan, we propose to use the background performance with low computational complexity due to the matrix obtained during the processing of the first hypercube use of random selection and prediction [10]. to calculate the matrix of anomalies on the three remaining Hyperspectral data are considered as a set of band hypercube data. The resulting 4 maps of anomalies in nadir vectors: scans are further combined into one general map to increase the reliability of the detection of anomalous objects and reduce the frequency of false detections. The anomaly where M - is the number of bands, and N - is the number of detector is also used for hyperspectral data obtained by the pixels. The background matrix and the anomaly matrix are second channel of the hyperspectral camera. As a result, we denoted as B  HMN and S  HMN. Anomaly detection get 5 different anomaly maps. The general scheme of data consists of separating the anomaly from the background, so processing by an optical-digital complex is shown in Figure the matrix of the corrected hyperspectral image Y can be 6. Fig. 6. Data collection by optical-digital complex in the longitudinal and transverse directions. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 151 Image Processing and Earth Remote Sensing In turn, a cloud of points is built from high-resolution hilly or mountainous areas, where the use of minesweepers images obtained by a camera of the visible range from is difficult. different angles using three-dimensional reconstruction algorithms of a digital terrain model. These point clouds REFERENCES are classified into several areas corresponding to local and [1] Why Landmines Are Still a Problem - International Campaign to global slopes using the spatial classification method, which Ban Landmines (ICBL) [Online]. URL: http://www.icbl.org/en- determines the class of a group of points based on its gb/problem/why-landmines-are-still-a-problem.aspx (05.10.2019). spatial position, shape, and size. Medium shear clustering [2] A. Khamis, “Minesweepers: Towards a Landmine-free World,” IEEE Canada International Humanitarian Technology Conference is used to perform spatial segmentation. This method is a (IHTC), 2015. non-parametric clustering technique that does not require [3] N.T. Thanh, H. Sahli and D.N. Hao, “Infrared Thermography for knowledge of the number of clusters or the shape of these Buried Landmine Detection: Inverse Problem Setting,” IEEE clusters [11,12]. The mean shift algorithm groups the point Transactions on Geo-science and Remote Sensing, vol. 46, no. 12, cloud into separate clusters, shifting the mean value to a pp. 3987-4004, 2008. denser direction, where most of the points are located. [4] P.A. Jacobs, “Thermal Infrared Characterization of Ground Targets These clusters are fed to the dispersion-based segmentation and Backgrounds,” SPIE Tutorial Texts in Optical Engineering, vol. TT70, p. 190, 2006. method, which in turn further segments the previously obtained clusters [13-17]. [5] E.M. Winter, M.A. Miller, C.G. Simi, A.B. Hill, T.J. Williams, D. Hampton, M. Wood and J. Zadnick, “Mine detection experiments Segmented point cloud data and multi-angle anomaly using hyperspectral sensors,” Detection and Remediation Technologies for Mines and Minelike Targets IX, Proc. SPIE, vol. maps are fed into the neural network [14], where, at the 5415, 2004. training stage, the corresponding pixels from the multi- [6] J.E. McFee, C. Anger, S. Achal and T. Ivanco, “Landmine angle anomaly maps are assigned to each class of local and detection using passive hyperspectral imaging,” Chemical and global slope. After the error in each neuron is less than the Biological Sensing VIII, vol. 6554, 2007. predetermined tolerance, the neural network is considered [7] M. Gagnon, P. Lagueux, J.P. Gagnon, S. Savary, P. Tremblay, V. to be trained. In the end, the work of the neural network Farley, E. Guyot and M. Chamberland, “Airborne thermal infrared creates a three-dimensional map of the anomalies, which hyperspectral imaging of buried objects,” Electro-Optical Remote Sensing, Photonic Technologies and Applications IX, vol. 9649, are compared with the terrain. From the data obtained at 2015. the early stages of processing, a three-dimensional model [8] A.N. Grigoriev, “Topographic radiometric correction of space of the surface of the visible range is constructed, which in hyperspectral imagery data,” Abstracts of the reports of the turn is combined with the obtained three-dimensional map scientific and technical conference "Hyperspectral devices and of anomalies, which are highlighted for the operator with technologies", Krasnogorsk, Russia, pp. 130-132, 2013. bright color. [9] A.D. Golovin and A.V. Demin, “Optical-digital complex for detection of remote mines and mapping of minefields,” Journal of IV. CONCLUSION Physics: Conference Series, vol. 1096, no. 1, pp. 012-011, 2018. [10] T. Zhou and D. Godec, “Randomized low-rank & sparse matrix An approach to the implementation of remote search decomposition in noisy case,” Proceedings of the International and mapping of hidden land and subsurface objects - anti- conference on machine learning, USA, Bellevue, WA, vol. 46, no. personnel mines, improvised explosive devices, and 12, pp. 33-40, 2011. unexploded ordnance, buried into the ground at a shallow [11] H. Woo, E. Kang, S. Wang and K.H. Lee, “A new segmentation depth, on the territory with different slopes using an method for point cloud data,” International Journal of Machine optical-digital complex based on multi-view long-wave Tools and Manufacture, vol. 42, no. 2, pp. 167-178, 2002. infrared hyperspectral scanning together with a three- [12] M. Fashing and C. Tomasi, “Mean shift is a bound optimization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, dimensional terrain reconstruction is proposed. The vol. 27, no. 3, pp. 471-474, 2005. method allows for increasing the reliability of detection by [13] D. Comaniciu and P. Meer, “Mean shift: a robust approach toward eliminating the relief and spectral distortion. The feature space analysis,” IEEE Transactions on Pattern Analysis and application of the anomaly detection method in multi-view Machine Intelligence, vol. 24, no. 5, pp. 603-619, 2002. hyperspectral images based on the decomposition of the [14] A.V. Mingalev, A.V. Belov, I.M. Gabdullin, R.R. Agafonova and hyperspectral image matrix into the background, including S.N. Shusharin, “Test-object recognition in thermal images,” spectral vectors of main ground objects in the image scene Computer Optics, vol. 43, no. 3, pp. 402-411, 2019. and an anomaly matrix containing spectral vectors of small [15] B. Xiang, J. Yao and X. Lu, “Segmentation-based classification for and unlikely ground objects, provides good performance 3D point clouds in the road environment,” International Journal of Remote Sensing, vol. 39, no. 19, 2018. with low computational complexity. A method of [16] L.I. Perlovsky and M.M. McManus, “Maximum likelihood neural combining local slope clusters classified from points networks for sensor fusion and adaptive classification,” Neural clouds of terrain with a segment of anomaly map of one of Networks, vol. 4, no. 1, pp. 89-102, 1991. the hyperspectral images taken from different angles is [17] Yu.V. Vizilter, V.S. Gorbatsevich and S.Y. Zheltov, “Structure- proposed. The technique allows excluding to a minimum functional analysis and synthesis of deep convolutional neural both spectral and spatial topographic distortions inherent in networks,” Computer Optics, vol. 43, no. 5, pp. 886-900, 2019. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 152