Landmines and cluster munitions are the main obstacle to the development of societies and the return to normal life after a ceasefire. According to the latest statistics [1], 80% of landmine victims are children who have nothing to do with the war or its causes. Therefore, it is necessary to prohibit the use of this type of blind weapon. Efforts to ban landmines have already begun, and we have an international campaign to ban landmines and cluster munitions with a number of signatory countries.
One of the earliest and most widely used methods is the metal detector. Thanks to the phenomena of electrical induction, this type of detector is able to detect objects containing metal underground. The most widely used mine detection method follows the same methods developed during World War II and directly involves people. A typical explosive ordnance engineer toolkit today is very similar to those used more than 50 years ago (it consists of a metal detector and a probe). However, this method has a number of disadvantages, namely low mine detection speed, high sensitivity to noise, and a short detection radius, which leads to the need for the operator to be physically present in close proximity to explosive objects.
Modern mine detection methods focus on increasing the detection radius and expanding the types of mines they can detect. Each method is suitable for detection in specific conditions depending on the type of mine casing, explosive, and soil.
Generally, most mine detection methods consist of three main components: a sensor to capture the signature of the mine, a signal or image processing unit to organize the received data, and a decision unit to determine whether a mine exists or not. Among the sensors, the following main types are distinguished: electromagnetic, acoustic, nuclear, biological, chemical and mechanical. Electromagnetic, namely electro-optical, sensors allow detecting objects at a great distance, and do so faster than most other types of sensors. Among electromagnetic sensors, hyperspectral sensors ProfIT AI 2024: 4th International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI 2024), September 25-27, 2024, Cambridge, MA, USA : svm@nau.edu.com (V. Sineglazov); lesogor.kirill@gmail.com (K. Lesohorskyi); immun56@gmail.com (V. Lytvynenko) : 0000−0002−3297−9060 (V. Sineglazov); 0000-0003-2773-7398 (K. Lesohorskyi); 0000-0002-1536-5542 (V. Lytvynenko)
are the most suitable for solving the problem of mine detection, since they have a high number of channels that allow detecting mines, while providing a good balance between dimensionality, the amount of useful information, and the speed of the intelligent system. A comparison of various electromagnetic sensors is presented in Table 1. As such, hyperspectral imagery is used for it's balance of information and redundancy. This method makes possible to measure the proportion of light reflected in hundreds of wavelengths at each image unit (pixel). Thus, we obtain a hypercube consisting of two spatial dimensions and a third dimension that contains spectral information. This method is used in the field of remote sensing for various purposes such as mapping, agriculture, astronomy, food monitoring, surveillance and others.
When light hits an object, it is either absorbed or reflected. The fraction of light that is reflected depends on the size of the molecules of the object that is reflected, the intermolecular distances, and the wavelength of the radiation. Each material, consisting of different components, can reflect light of different wavelengths differently. There are several approaches to target detection using hyperspectral imaging: some are supervised, where the spectrum of the data is known in advance; others are unsupervised, based on finding targets that are spectrally different from their surroundings. The latter type of information does not require knowledge of the target spectrum in advance. However, this type of detector is characterized by a high rate of false alarms, since rare events in the image that differ from their background will be labeled as targets.
Spectral image processing is a non-trivial task and has a number of challenges that need to be taken into account to create an effective algorithm. The first problem is the high dimensionality of the data. Hyperspectral images cover hundreds of spectral bands, which increases the size of the data and creates the curse of dimensionality. This complicates the process of creating models and increases the cost of computations required for both training and data processing. The second problem is the limited labeled dataset. Collecting and annotating hyperspectral data is a timeconsuming and expensive process, which often results in a limited amount of available training data. This limits the ability to effectively train complex models. The third problem is the coupling of spatial and spectral information. Spectral features are key for material recognition, but spatial features such as texture and context are also important for accurate classification. Efficiently integrating these two types of information is a challenge for classical computer vision algorithms. The final challenge is computational power needed for hyperspectral data processing, especially when using modern deep learning methods. This limits the ability to use complex models in real time.
This paper is devoted to developing a comprehensive landmine detection method with a novel dual-mode (batch / real-time) capability that utilizes a hyperspectral sensor in conjunction with a neural network classifier to detect the landmines. The proposed method addresses the problem of high dimensionality by applying a two-step preprocessing to reduce the dimensionality while maintaining both spectral and textual information. Proposed approach provides a flexible tool to explosive ordnance removal teams that supports initial area survey with batch mode and enables mine removal operations support with the real-time mode.
This section provides an overview of past projects using hyperspectral imagery. Note that some military studies and projects may exist in this area, but they are not considered here due to lack of information. One of the first projects to study mine detection using infrared wavelengths was conducted at Defence Research & Development Canada (DRDC). DRDC began its research in support of the Canadian Army on mine and unexploded ordnance detection in 1978 and in collaboration with Itres Research on hyperspectral imaging for mine detection in 1989. The algorithms developed during this project can be applied to pre-processed images from hyperspectral imagers. An early project proposed a hierarchical image processing algorithm to detect a sparsely distributed bright region a few pixels wide in a monochromatic image [2]. The pre-processing operation is performed to remove distortions, dropouts, overlapping regions, misregistration and any other artifacts and defects. Unsuspected regions are discarded to reduce the data size. Then, the suspected regions are segmented into homogeneous subregions and the morphological features of the subregions are extracted. Based on the extracted features, the subregions are classified. Finally, the spatial relationships between the mine-like objects are determined. The supervised method analyzes these relationships and classifies the regions as a minefield by providing a certain likelihood ratio.
Fusion of visible and SWIR bands can provide better detection results. Basic fusion of two spectral bands provides acceptable segmentation of objects from the background, regardless of the illumination conditions. In other words, choosing a set of two or three spectral bands from an image has been shown to be as effective in differentiating artificial objects from the background as using all spectral bands simultaneously [3]. Such fusion has the potential to detect mine-like objects in an image using an integrated camera with visible and SWIR sensors and more sophisticated and specialized detection algorithms. [4] describes a Defense Advanced Research Projects Agency (DARPA)-sponsored experiment to test the feasibility of detecting buried mines using midwave infrared (MWIR) (3 to 5 μm) and longwave infrared (LWIR) (8 to 12 μm) hyperspectral bands. The project focuses on detecting surface disturbances caused by buried mines. Previous experiments have shown the ability of VNIR and SWIR imagers to detect surface disturbances [5], [6], [7]. However, the problem was the high rate of false alarms caused by surrounding vegetation and rocks. According to the authors, the main rationale for detecting buried mines using spectral properties is that the surface properties are somewhat different from the properties of the subsurface soil. The impact of soil on the surface changes some of its physical and chemical properties. These experiments showed that spectral information is necessary for mine detection. The objective of the study presented in [8] is to test the design of multispectral and hyperspectral imagers that are able to achieve better detection performance while meeting the requirements and conditions of target detection. Target detection requires detection in both day and night conditions. Panchromatic or multispectral images in the VNIR and SWIR ranges provide this capability during the daytime. However, for military use, the MWIR and LWIR bands are needed for nighttime operation. Due to the high correlation of spectral bands of background materials in all background conditions, the target detection capability is high using the MWIR and LWIR bands. After testing the two bands in the MWIR and LWIR bands, the authors concluded that thermal multispectral images would provide better target detection and false alarm rate than a single-band infrared sensor. The tests showed that properly selected small bands could provide good detection, the optimal band range being between 8 and 10.5 micrometers. The usefulness of using LWIR with MWIR increased significantly compared to using MWIR alone.
DSTL Defense Science and Technology Laboratory Mine Defense Project. A project similar to the DRDC and DARPA projects was started in Rythen to detect mines using a VNIR imager [9]. The program was called the DSTL Mine Defense Project. Using a tripod-mounted VNIR SOC 700 hyperspectral camera, the team captured high spatial resolution images of the mines. However, the data is mainly used to explore different processing methods rather than to evaluate the PD and FAR of the sensor. To process the data, the authors used principal component analysis (PCA) for dimensionality reduction and anomaly detection for classification. The authors avoided using spectral comparisons between the target and each pixel in the image, as this would be very timeconsuming due to the low target-to-background ratio. The results were still preliminary, but the authors concluded that VNIR can distinguish surface mines from the background. In India, researchers proposed a hierarchical algorithm for mine detection using infrared images, which consists of pre-processing (contrast enhancement -filtering -smoothing), segmentation, feature extraction, and ANN-based classification [10]. The authors tested the algorithm on surface mines in two soil types: black cotton and sand. During pre-processing, the image is converted to gray color. The two most important preprocessing steps are contrast enhancement and noise removal. During the tests, the authors used a small NN with 1 hidden layer and 4 neurons. The results obtained on a simple dataset are good, but it is not expected that the algorithm will work well on another field or soil type, since the data used in the training stage is not complete enough. In 2015, TELOPS, a Canadian research company specializing in infrared and hyperspectral imaging, demonstrated the feasibility of detecting buried objects using an airborne LWIR hyperspectral imager [11]. From the aircraft, they obtained thermal hyperspectral images of areas that contain previously buried artificial objects. They found that the disturbed soil directly above the buried target is warmer than the undisturbed soil area next to it [11]. Comparing the emissivity data obtained by decoupling temperature and emissivity, buried targets are displayed as part of the hottest ground region within the scene, but additional classification or information is needed to distinguish buried objects from other naturally hot regions. It is worth noting that most of the works reviewed pay much attention to the quality of electro-optical sensors and pre-processing, but the classifier itself is quite simple. The use of modern artificial intelligence technologies, namely convolutional neural networks (CNN), will have a positive effect on both the overall accuracy and specificity of the classifier. Another common problem is the limited training dataset. Semi-supervised learning methods are used to solve this problem.
In this section, we present the detection algorithms used to detect targets in hyperspectral imagery. In addition, we present several pre-processing steps and hyperspectral data processing typically used in the pre-processing step to facilitate further detection or classification. An overview of the various processing methods used for data fusion, spectral unmixing, classification, and target detection can be found in [12].
The first step of pre-processing is normally a contrast enchantment. The process of image contrast enhancement consists of a set of methods that attempt to improve the visual appearance of an image or transform the image into a better form suitable for human or machine analysis [13]. Image contrast enhancement methods are divided into two main categories: spatial domain methods and frequency domain methods. Spatial domain methods are applied directly to the pixels of an image. In frequency domain methods, the image is processed in the frequency domain after applying the Fourier transform to the original data. Contrast enhancement is one of the most commonly used image contrast enhancement methods. In the case of mine detection, the role of contrast enhancement is to enhance the difference between the mine and the background materials [14]. The main contrast enhancement methods used are histogram equalization and morphological contrast enhancement.
Contrast enchantment is followed by filtering step. Filtering is an operation that always reduces noise or shapes blurred areas in an image to make it clearer and more suitable for further processes. In hyperspectral image filtering, several methods commonly used in image processing have been upgraded to achieve multi-channel reconstruction. There are two groups of filters: one is based on the assumption that intra-channel information is separable from inter-channel information, i.e., spectral and partial information are separable. These filters are called hybrid filters. In this case, the first step is to decorrelate the channels of the Fourier transform or PCa, and then apply a classical 2D reconstruction method such as the Wiener filter or the static wavelet transform. The other group consists of several proposed filters that do not rely on the assumption of spectral and spatial separability [15].
After filtering, segmentation is performed. In the remote sensing community, segmentation is defined as the process of finding homogeneous regions in an image followed by classification of these regions [16]. In image processing, there are many methods used for segmentation, however, not all of them are applicable to multispectral and hyperspectral images. Some methods, such as watershed algorithms, have been modified for hyperspectral image segmentation. Globally, segmentation algorithms are divided into two categories: edge-based and region-based. Edge-based methods detect the boundary using the discontinuity property. In the region-based algorithm, the pixels in a region are grouped using the similarity property. In the following, we introduce the main methods used in hyperspectral image segmentation.
The next step is feature extraction from the segmented image. Feature extraction is the transformation of data from a high-dimensional space to a lower-dimensional space chosen in such a way as to preserve as much of the information of interest in the data as possible. Feature extraction is used in hyperspectral image analysis to address the problem of the small number of training data samples compared to the high spectral resolution of the image and to reduce computation time. There are many feature extraction algorithms that have been introduced; some are linear while others are nonlinear. When working on mine or target detection, not all feature extraction algorithms are useful because the targets of interest are usually rare and feature extraction can highlight key features of the target. The most common feature extraction algorithms are Principal Component Transformation (PCT), Linear Discriminant Analysis (LDA), matched pursuit [17], neighborhood embedding [18], Sammon's mapping [19] and nonparametric weighted feature extraction [20], Artificial neuron Networks (ANN).
Hyperspectral image classification is one of the key tasks in remote sensing, which involves identifying the types of materials or objects on the earth's surface based on their spectral characteristics. Hyperspectral images contain information in hundreds of narrow spectral bands, which allows obtaining detailed data on objects and materials that cannot be detected by traditional multispectral images.
In this work, image classification is considered in the supervised setting. Formally, a labeled dataset 𝐷 = {(𝑥 ! , 𝑦 ! ), … (𝑥 " , 𝑦 " )}, where 𝑥 ∈ 𝑋, 𝑦 ∈ 𝑌, 𝑋 is the input space, 𝑌 is the label space is used. The goal of supervised learning is to train an approximator function 𝑓(𝑥, 𝜃) ≈ 𝑃(𝑦 |𝑥), 𝑥 ∈ 𝑋, 𝑦 ∈ 𝑌, where 𝑓 -is the approximator function, 𝑃(𝑦 |𝑥) -is the target marginal distribution, 𝜃 are the approximator parameters.
For hyperspectral images, each point in the input space 𝑋 is a tensor (hypercube) with the dimensions of 𝐻 × 𝑊 × 𝐶, where W is the image width, H is the image height, C is the number of image channels. The fundamental difference from classical computer vision tasks is that the dimension C is in the range [100, 1000], which significantly increases the number of input parameters. Thus, an image in the RGB spectrum with a resolution of 256×256 pixels contains 196608 parameters, while a hyperspectral image of the same resolution using 100 channels already contains 6553600 parameters, which significantly increases the requirements for the efficiency of image coding for effective processing.
As the problem is considered in the multi class classification setting, the output (or label) space is described as a set 𝐶 = {𝑐 ! , … 𝑐 # }, where k is the number of distinct classes. Approximator f is trained to correctly assign a label 𝑐 ∈ 𝐶 for each possible input 𝑥 ∈ 𝑋 in the input space, generalizing unknown samples.
In this work we propose a three-stage approach to hyperspectral image classification that combines classical pre-processing techniques and more robust ANN-based classification methods. The three steps are:
1. Normalization; 2. Filtering and texture extraction; 3. Feature extraction and classification.
In the following subsections we describe each step in more details.
The first step of the method is a preprocessing pipeline that reduces the dimensionality of the input, normalizes the data and prepares it for feature extraction and classification step. The first step of pre-processing is noramlziation. In this step, the spectral image is normalized and filtered. First, all spectral data is normalized channel by channel using min-max normalization:
where c -is the spectrum channel, 𝑖 ∈ 𝑊 -is the pixel's horizontal coordinate, 𝑗 ∈ 𝐻 -is the pixel's vertical coordinate. This helps to normalize values, and by extension spectras, which simplifies further processing. Min-max normalization was chosen, as it is well suited to normalize the data for all of the channels while maintaining local patterns within each pixel's spectral bands. The next step is filtering using the histogram method Local Binary Patterns (LBPs). This is a spatial filtering method that is used to extract spatial features, especially textures, which significantly increases the classification accuracy. LBP adjusts the intensity value of each pixel by applying a transformation function to the neighborhood function. First, a neighborhood function is selected. Often, the Moore neighborhood function is used, but other neighborhood functions can be used to increase the texture receptive field. For each pixel, a vector of texture features is calculated:
where 𝑔 1 -is the neighborhood pixel value, P -is the neighbourhood space, 𝑔 1 ∈ 𝑃, 𝑔 ' -is the central pixel. After the iamge is translated into LBP encoding, they are used to build a texture's histogram. LBP's biggest advantage is high processing speed and the ability to preserve spatial patterns (and textures) for high-resolution landmine detection. After the histogram is created, an additional filtering stage is performed. In this stage, a morphological filtering is used to remove noise and speckles form the image. Morphological filtering allows you to filter out noise and insignificant objects from the texture. Morphological image processing is a set of tools for analyzing and processing the structural features of images based on set theory. These methods allow you to highlight and enhance the spatial characteristics of objects in images, which makes them extremely useful in image processing and computer vision. The first stage is erosion -reducing the number of objects in the image by removing pixels on the boundaries of objects. This removes insignificant noise:
(𝐴 ⊖ B)(𝑖, 𝑗) = min
where A -is the original histogram, B -is a structural element. After this, an extension stage is performed. During this stage, the objects are diluted, and effectively extended to their original size by adding extra pixels on the edge of the bright objects as follows:
where A -is the original histogram, B -is the structural element. Morfological filter is created by composing erosion and dilation functions, which removes noise from the histogram, which makes object's boundaries sharper:
(5) This filter is then applied to the texture histogram generated by applying formula (2) after the data normalization and is then used as a basis for feature extraction and classification. This filter works great for landmine detection as it is fast and creates a great contrast between target objects (landmine) and surrounding pixels, while removing noise around the edges. The disadvantage of using this approach is that in high-density minefields this method can merge several mines that are nearby into a single entity, increasing the area that has to be manually surveyed."
Convolutional neural networks are used to solve the problem of feature extraction and classification. This is a common approach for solving computer vision problems. There are many architectures, but most of them are designed to process images with a small number of channels [29]. Due to the high dimensionality of hyperspectral images, known architectures are poorly suited for processing such images in their original form. However, after pre-processing, the dimensionality and complexity of the data is significantly reduced, which allows to synthesize [30] a simpler architecture [31]. The proposed network consists of the following types of layers:
1. convolution layer is the basic building block of a CNN, where the convolution operation occurs. Filters (kernels) slide over the entire image, calculating the dot product between the filter and a portion of the input image, creating feature maps; 2. pooling layer performs the operation of reducing the dimensionality of feature maps, preserving the most important features. The most common types are MaxPooling (selects the maximum value in each window) and AveragePooling (selects the average value); 3. BatchNorm layer is used to normalize feature maps, which improves stability and learning speed; 4. dropout layer is used to prevent overfitting by randomly "switching off" some neurons during training; 5. fully connected layer has it's neurons connected to all neurons of the previous layer, which enables combining features and making a final decision;
ReLu is used as the activation function, and Cross-Categorical Entropy Loss is used to train the neural network. Network architecture is presented in Figure 1.
The proposed method can be applied in two modes -real-time and batch processing of a largearea hyperspectral cube. The batch mode provides higher overall speed and accuracy due to access to larger computing resources, but requires the agent to completely scan the search area. The realtime mode provides information much faster, but increases energy costs on the part of the agent, which can negatively affect the overall speed of zone processing, since the agent will spend more time recharging. Schematically, batch processing is shown in Fig. 2 (a), and processing in batch mode is shown in Fig. 2 (b). The training phase is common for both approaches. For training, hyperspectral images of the search areas are collected and then labeled. For labeling, both binary classification (mine present -no mine) and multi-class classification can be used to determine a specific mine type. After labeling, the hyperspectral images are normalized, filtered and divided into small sections using sliding window algorithms with a small intersection between each section. After splitting, the dataset is used to train the model using supervised learning.
In batch mode, the agent performs a full scan of the area of interest, obtaining a hyperspectral image of a large area. After that, the data is also normalized, filtered and split into several sections using a sliding window algorithm. Each of the sub-sections is processed using a convolutional neural network. The classification results are combined with the telemetry collected by the agent to create a geoinformation database.
In real-time mode, classification and database creation are performed directly on the agent during the scan. This allows you to analyze a part of the area much faster, but in this case the algorithm itself is much less efficient (since normalization and filtering are performed on slices with a high percentage of intersection). Also, the overall scanning time of the area increases, since the algorithm requires high energy costs, which results in the need for additional recharges.
The algorithm was tested on the Pavia University hyperspectral dataset [27], which was injected with synthetically generated signatures of various mines. To emulate more realistic camouflaged mines, we used a blend operator to weightedly average the spectral signature of the mines with the surrounding environment.
This dataset was collected using an aircraft-mounted hyperspectral sensor and contains images of an urban area near the University of Pavia in Italy.
The dataset has a resolution of 1.3 meters per pixel and has 103 spectral channels ranging from 430 to 860 nanometers. These channels cover the visible and near-infrared spectra, providing high spectral resolution for identifying different materials and objects. The dataset includes 9 classes: asphalt, lawn, road, trees, soil, bitumen, tiles, shadow, buildings. An additional class, mine, was introduced into the dataset in a ratio of 5-95% percent, replacing some of the existing class labels soil, lawn, road, shadow. During training, the dataset is split into a training and validation dataset in a ratio of 80-20%. During training, the Adam optimizer [28] is used, the learning rate is set to 0.01, with a subsequent decrease to 0.001. Recommendations outlined in [31,32] 2. Confusion matrix is presented in the Table 3. An example of the original and classified hyperspectral image is shown in Fig. 3.
The results obtained in this experiment are promising. They indicate a viability of applying the proposed method in the real world scenario. The proposed model was able to achieve high throughput in the batch mode, and adequate inference time in real-time mode. Average accuracy of 87.5% is somewhat low, however in this setting the network was trained to classify not only landmines, but other classes as well, which brought down the overall results. Confusion matrix, however, indicate a decent classification accuracy. The most important aspect is recall, as false negative errors are extremely dangerous when dealing with explosive ordnance. The proposed method was able to achieve 100% recall, which means that no landmine was undetected. However, precision metric is 86.33%. While this is not critical on the scale the test was performed (as only 19 false positives would have been checked), real minefields can have over 1000 mines planted, which would drastically slow down mine removal process.
While results are promising, it is important to note that used dataset has severe limitations. The key limitation is the granularity of source dataset, which somewhat skews the precision and recall metrics. Higher-granularity dataset is unlikely to impact recall metric as long as decision boundary is set reasonably, however will greatly increase false positive rate near the edges of the landmines.
Another major limitation of the experiment is that synthetic dataset does not give realistic recall metrics, as they can depend on many factors, such as the type of soil, time of day, ambient temperature, landmine depth, landmine material, atmospheric pressure. A purpose-built dataset which can control these parameters are required to validate the experimental results before the method can be applied in real-world landmine removal operations.
This paper presents a hybrid method for detecting mines in hyperspectral images that combines the strengths of various existing methods. The proposed method uses an integrated approach to preprocessing and filtering the obtained data, which significantly simplifies the feature extraction and classification step. The preprocessing step made it possible to significantly simplify the architecture of the convolutional neural network of the classifier, which has a positive effect on the accuracy and speed of both training and inference. The proposed method operates both in batch mode and in real time (at ~13 FPS), which expands the practical possibilities of its application.
The main limitation of this work is the use of a sensor with a resolution of 1.3 meters per pixel, which is not sufficient to accurately detect the location of mines. Also, in the synthetic dataset, the noise was not so pronounced due to the fact that the mines were introduced using the blend operator.
Further research can be developed in several directions. Firstly, obtaining a real landmine dataset with a higher resolution will allow testing the proposed algorithm in conditions close to real world. In this paper, we considered a multi-class classification problem where each grid segment could belong to 10 classes, but only 1 of them could be a mine. A more interesting scenario is the absence of irrelevant classes with several different types of mines. In this scenario, the sensitivity of the classifier will be overstimulated, which will lead to a high level of type I errors. Also, the classifier created using the current method has poor generalization ability with respect to the soil in which the mines are buried. When using this method in a location different from the one in which the training dataset was collected, a drop in accuracy is expected. Improving the generalization ability of the classifier without losing accuracy and increasing sensitivity to false positives is an open problem.
Based on the factors, outlined above, future research will be concentrated on acquiring a real landmine dataset to validate and calibrate the existing approach. By introducing multiple types of landmines, it will be possible to properly calibrate the decision thresholds for the classifier to decrease it's sensitivity to noise and increase precision.