This paper concerns the problem of hyperspectral imagery multi-class classification. A hybrid three step iterative method is proposed. The method consists of pre-processing, training and proxy-labeling steps. During the pre-processing step the data is normalized and filtered to minimize the noise and smooth the data. The second step involves training a convolutional neural network to train a discriminator. Third step utilizes the discriminator from the second step to label high-confidence samples from the unlabeled pool. An overview of each component is given. The efficiency of dimensionality reduction is evaluated. Results indicate that dimensionality reduction step both speeds up the subsequent learning and improves the efficiency of semi-supervised learning.
Hyperspectral imaging is an advanced remote sensing method that collects detailed
information about the earth's surface. It projects light waves in hundreds of narrow spectral
ranges, providing a rich dataset for identifying and characterizing materials, assessing
biophysical parameters and monitoring environmental conditions. A number of studies are
devoted to the use of supervised learning for hyperspectral classification. The most common
task for hyperspectral image processing is multi-class classification. This is a simple, yet
effective way to model connections between a discrete target variable and a set of input data.
There are a number of problems associated with the use of data for hyperspectral classification:
high dimensionality of hyperspectral data - hyperspectral data consists of hundreds of
light spectrums, which complicates the development of accurate models, which leads to
the curse of dimensionality [
Semi-supervised learning, on the other hand, allows utilization of a combination of labeled and unlabeled data to train the model, achieving higher accuracy than a normal supervised learning.
When considered under the semi-supervised framework, hyperspectral image classification in it’s core is similar to other multi-class classification tasks. More formally, we start by introducing a labeled dataset L={( x0 , y0) , … ,( xn , yn ) }, where x∈ X , y∈ Y , X is the input space, Y is the label space, n is the size of labeled dataset. It is complemented by unlabeled dataset U ={ xn+1 , … , xn+m }, x∈ X , m is the size of the unlabeled dataset. A combined dataset D=U ∪ L is used to train model parameters θ of an approximator function f ( x , θ ) ≈ y’. The approximator function is learnt to minimize a loss function and make predicted labels as close to the groundtruth labels as possible.
Training is an iterative process θi , Li , U i=T ( θi−1 , Li−1 , U i−1 ), where i is the training iteration (or epoch), T is a training step. Depending on the internal implementation of the algorithm, labeled and unlabeled data sets can be updated between the training iterations. Hyperspectral imagery has a lot in common with classic computer vision tasks – the data is represented as a 3-dimensional cube of data x H ×W ×C∈ X , where x is the data sample, H is the image height, W is the image width, C is the number of channels. However, unlike conventional image recognition, however, the number of channels C is usually in the range of 100-500, instead of the normal 3-4 channels. This leads to increase in the input size, e.g. for an input with H = 48, W = 48 the normal configuration (C = 3) would result in 6912 features. Hyperspectral images of the same resolution with C = 100 would have 230400 features, an increase of ~ 33 times, which can be even larger in practice.
Semi-supervised framework with deep neural networks is able to address the issues of sample size and high dimensionality, however the problem of noise is still present. As such, hyperspectral images must be pre-processed to reduce noise and remove artifacts. The preprocessing is modeled as a function P ( x H ×W ×C )= x ' H ×W ×C , x X , x ' X , where x is the original input, x’ is pre-processed input.
A general framework for hyperspectral multi-class classification consists of two stages: data pre-processing and discrimination. Pre-processing steps may vary depending on the discriminator used. Recent research uses the variety of discriminator models, such as SVM, matched filters, and neural networks. Among these methods, neural networks is the most robust as it allows to efficiently learn the low-dimensionality embeddings from the highdimensionality dataset.
Despite widely acknowledged danger and economic damage mines are known to posses [
As such, a variety of techniques were considered for remote probing. Hyperspectral
imagery[
More recent approaches leverage artificial neural networks to detect landmines more reliably. In India, researchers proposed a hierarchical algorithm for mine detection using infrared images, which consists of pre-processing (contrast enhancement - filtering smoothing), segmentation, feature extraction, and ANN-based classification [9]. The authors tested the algorithm on surface mines in two soil types: black cotton and sand. During preprocessing, the image is converted to gray color. The two most important preprocessing steps are contrast enhancement and noise removal. During the tests, the authors used a small NN with 1 hidden layer and 4 neurons. The results obtained on a simple dataset are good, but it is not expected that the algorithm will work well on another field or soil type, since the data used in the training stage is not complete enough. In 2015, TELOPS, a Canadian research company specializing in infrared and hyperspectral imaging, demonstrated the feasibility of detecting buried objects using an airborne LWIR hyperspectral imager [10]. From the aircraft, they obtained thermal hyperspectral images of areas that contain previously buried artificial objects. They found that the disturbed soil directly above the buried target is warmer than the undisturbed soil area next to it. Comparing the emissivity data obtained by decoupling temperature and emissivity, buried targets are displayed as part of the hottest ground region within the scene, but additional classification or information is needed to distinguish buried objects from other naturally hot regions.
It is worth noting that most of the works reviewed pay much attention to the quality of electro-optical sensors and pre-processing, but the classifier itself is quite simple. The use of modern artificial intelligence technologies, namely convolutional neural networks (CNN), will have a positive effect on both the overall accuracy and specificity of the classifier. Another common problem is the limited training dataset. Semi-supervised learning methods are used to solve this problem.
Data preprocessing is an important step in the data analysis process to ensure its accuracy, reliability, and readiness for further analysis. In this work we use a 4-stage pre-processing framework to minimize the impact of the noise and decrease the dimensionality of the input.
Normalization is the basic technique [11] aimed at adjusting the flow of changes in lightning, atmospheric conditions and other factors on the spectral data. The first step is to align the values of different bands as their reflectance values can be highly different. This is achieved by applying a min-max[12] normalization. Min-max normalization is a common technique to normalize multi-band images, be it hyperspectral or simple RGB and is defined as:
xi , j−min x x 'i , j= max x−xHm,Win x xH,W xH,W where x 'i , j is the normalized pixel value, xi , j is the original pixel value, mxHa,Wx x is the maximum intensity value and min x is the minimal intensity value within the channel. Another xH,W common normalization technique to stabilize the training is spectral normalization [13]. Although it is a powerful tool, it’s primary use case is the stabilization of generative models[14], and as such it is ill-suited for discriminator training. The removal of noise may result in the influx of unwanted signals and artifacts that may interfere with the results of the analysis[15]. A variety of filters, such as a median filter [16] for removing impulse noise:
x 'i=median [ xi−n , xi−n+1 , … , xi , … , xi+n−1 , xi+n], (2) where x ' is filtered signal, x is the original signal, median is the median operation. Median filters are linear, and often are defined via kernels in case of multidimensional data. A Gaussian filter [8] is used for smoothing data to decrease the interference (in this work we utilize a twodimensional version of the filter):
G (i , j )= e−(i2+ j2)/2σ2 where i,j are dimensions along which the filtering is performed,σ is the standard deviation of the distribution. This filter is also used only in one band to decrease the inference.
Depending on the approach, the next step is often skipped, however it can have a great impact on the discriminator’s performance, as it is shown that higher dimensionality leads to rapid decrease in discriminator’s accuracy and exponential increase in model’s parameters [17]. As such, dimensionality techniques are often applied, especially for higher-dimensional data. Some of the bands may not be important for the analysis, and as such can be discarded to decrease the dimensionality. Principal component analysis (PCA) is a common technique [18], used to decrease the dimensionality of the data and identify the main components, which allows for the largest portion of information to be reduced and the influx of noise.
In this work we explore a class of problems that can be classified based on narrow, independent bands of data, as such linear methods are preferable due to their specificity. In these scenarios, linear dimensionality reduction methods are preferable as they are simpler and have comparable effect as nonlinear methods. However, for certain types of problems, it is beneficial to use non-linear dimensionality reduction methods, such as, Isomap[19], that perform better when noise forms non-linear cross-band structures.
Multiple types of discriminators are considered applied to the hyperspectral data classification. Several methods were used historically to process hyperspectral images, however neural networks are considered the state-of-the art approach. They are able to efficiently learn low-dimensional embeddings for hyperspectral data. Convolutional neural networks (CNN) are the most common approach to processing high-dimensional data, as convolutional layers are the most effective approach to reducing dimensions while minimizing the number of parameters. There are two types of convolutional layers - 3d and 2d convolutions which operate on the same principle: A discrete convolution on a finite set of values is determined using the formula
M ( f ∗g ) [ n ]= ∑ f [ n−m ] g [ m ], (4)
m=−M where f and g are some discrete functions, M is a matrix size, n, m are matrice’s values. If I (image) and K (kernel) are matrices, the two-dimensional convolution is determined using the formula where input ( N i , k ) is the k-th channel of the i-th training sample of size H, W for Conv2D and H, W, D for Conv3D; W – image width, H – image height, D – is the image depth (number of channels); C¿– number of input channels; N is the number of training samples in the batch; weight (C outj , k ) - convolution weights (kernel) between the k-th channel of the input and C outth channel at the output of the layer; bias ( C outj ) is convolution shift for the C outjth channel’s at the output of the layer, C out is the number of channels at the output of the layer.
However, apart from using the correct layers, it is also important to pick the appropriate architecture. ResNet [20] is state-of-the art architecture that utilizes classical CNN architecture with residual connections, which assist in gradient propagation along the network. ResNet is arranged as a series of blocks called bottlenecks. For hyperspectral data processing, we utilize a 3D version of ResNet [21], and introduce a more detailed architecture in section 4.
Loss functions play a key role in training deep neural networks because they determine exactly how the model evaluates its performance and adjusts its parameters.The loss function is a key component in the process of developing machine learning models, as it measures how well the model matches the given data. In this work, we use a hybrid method based on semisupervised learning, however we use a classical proxy labeling approach, and as such our loss does not have a semi-supervised term. Categorical cross-entropy loss is commonly used in training multi-class classifiers [22]:
C CE=−∑ ti log2 si, (7) i=0 where C is the number of classes, tiis the predicted probability, si is the groundtruth probability.
Semi-supervised learning allows the use of a combination of labeled and unlabeled data. In our case, proxy labeling [23] is the most straightforward approach that is easy to utilize as it defines all the parameters implicitly, with the minimal confidence threshold being the only hyperparameter of the algorithm.
The proxy labeling is an iterative process, where during each iteration a labeled dataset is extended by using the trained model to label unlabeled samples, extend the labeled dataset, and repeat the same process until either all of the labels are labeled or iteration limit is reached. More formally, proxy labeling consists of two stages - train (8) and update (9) that are repeated in the loop:
θi , bi=train ( f , θi−1 , bi−1 , Li−1) , (8) where θ are model weights, b are model biases, f is the approximator function, L is the labeled dataset, train is the training function. During the semi-supervised learning, datasets are updated between the training iterations as well as follows:
Li= Li−1∪ Si , U =U i−1 ∩ S , S={u|f (u , θi , bi) ≥ ϵ , u∈ U i−1}, (10) where L is the labeled dataset, U is the unlabeled dataset, f is the discriminator model, ϵ is the minimum confidence threshold,Wi, bi are discriminator’s parameters learnt on i-th iteration. This approach is based on the smoothness assumption (for two points x and x’ that are close to each other in the input space, their labels y and y’ should match), but does not require an explicit definition of distance metric, as it is instead replaced by the embeddings of the discriminator and confidence of the discriminator.
In this work we propose an iterative three stage approach to hyperspectral image classification. The first stage consists of image pre-processing, followed by the iterative proxylabeling approach to learn the features. The overall training flow is shown in Fig. 1.
In the first stage data is pre-processed by normalizing and filtering the original dataset. This stage sets up the dataset for future processing. The next step is dimensionality reduction through PCA. This step analyzes normalized data, identifies important bands and decreases dimensionality. Then, the training loop is engaged with continuous proxy-labeling training.
In this work we use a custom CNN architecture that is optimized[24, 25] to handle a large number of channels by squashing them in the first few convolutional layers with 3D layers. The network architecture is pruned using methods outlined in [26, 27] to ensure the best performance during both inference and training. The exact model architecture is shown in Fig. 2.
Semi-supervised learning utilizes a common proxy-label framework - we use 90% as the confidence for unlabeled samples to be transferred to the labeled pool. This update is performed once in 5 epochs of training. Once added to the labeled pool, the samples are removed from the unlabeled samples and treated as ground truth for all of the following training iterations.
It should be noted, that the proposed discriminator architecture is somewhat simplistic compared to the state-of-the-art networks, such as ResNet. The reason for that is twofold. On the one hand, is that the proposed approach only has to classify a fairly modest number of classes (either binary for mine - no mine, or multi-class with up to 20 classes if mines should be classified by their type). As such, the architecture can be simplified to decrease the computational capacity needed during the inference stage. The second reason is that decreased computational cost allows to run the system in a real-time capacity, which is useful for real-time surveying.
As such, the system can be operated in two modes – real-time and batch processing. In the batch mode, the agent first surveys a full area of interest with hyperspectral camera. The collected hypercube is then sliced with a sliding algorithm into a subareas that can be processed by the CNN. Each subarea is pre-processed and passed through the discriminator. Label data is then combined with the telemetry collected by the agent to create the geoinformational database that contains the information about the mines in the area. Unmanned aerial vehicle (UAV) is proposed as the agent, as it offers superior mobility and scanning throughtput [24]
In the realtime mode the data is available in the live feed from the agent. Inference is performed on the agent itself and can be stored on board and broadcasted to ground control station during the survey. In this case, the geoinformational database can be recovered once the mission is compelte to obtain a full map of the minefield. The general flow of both approaches is outlined in Fig. 3
The model was trained with the help of a stochastic gradient descent (SGD) optimizer with an input speed of 0.001, a batch size of 32, and a categorical cross-entropy loss function.
The main evaluation metric used to evaluate the classification efficiency of the model was accuracy.
The Indian Pines dataset is a hyperspectral dataset collected by the AVIRIS sensor over the Indian Pines test site in the southern state of Indiana. The University of Pavia dataset is a hyperspectral dataset collected by the ROSIS sensor over the University of Pavia, Italy.
Results from running the convolutional neural network (CNN) model on the Indian Pines and Pavia University datasets demonstrate the high efficiency of the model. For the Indian Pines data set, accuracy reached 85%, indicating that the model is good at recognizing different classes in folded hyperspectral images. For the University of Pavia data set, the accuracy was 87%, which is also a significant achievement due to the complexity of the data set. Results visualizations are present in the Fig. 4 and 5, tabulated data is presented in Table 1.
It is worth noting, that removing the dimensionality reduction step has a limited effect on the average accuracy, however, extra training epochs were required as not all of the unlabeled data was used and the classifier has not reached its peak discrimination capacity.
The model was tested in the batch mode, and the results were patched together to provide a view of the minefield. In batch mode we were able to achieve an average of 63.26 ms per inference,approximately 17 FPS, which makes the method applicable to real-time problems. 0.0031
The proposed multi-step hybrid semi-supervised neural network approach is efficient in processing of hyperspectral imagery. The approach is able to efficiently reduce the dimensions of the input data by discarding low-information bands in the pre-processing step via a PCA or other dimensionality reduction tools and then applying a neural network-based discriminator to the processed data to solve the problem of hyperspectral multi-class classification. A semisupervised learning loop based on proxy-labeling is used to augment the learning capacity, decrease the amount of labeled data needed, and minimize the number of the model’s hyperparameters.
The research has several limitations. Firstly, the PCA is a linear dimensionality reduction algorithm, which might remove information, especially in hyperspectral data, as non-linear structures are likely to be lost in the dimensionality reduction procedure. Additionally, used CNN architecture is fairly simplistic and might not have enough learning capacity. Lastly, proxy-labeling threshold is constant during the training process, which might lead to suboptimal labeling as the number of classes grows.
The approach is comparable to the state-of-the-art methods, however it is faster to train and adapt as it has fewer parameters to tune. The drawback of this approach, however, is vulnerability to poorly labeled data. It is also sensitive to non-linear noise structures, as PCA is more likely to remove informative bands in its presence. Further research includes two primary directions: improving pre-processing and optimizing semi-supervised learning. Pre-processing can be improved by sacrificing learning speed for non-linear dimensionality reduction methods, such as Isomap or Laplacian eigenmaps. Semi-supervised learning aspect would benefit greatly from defining an explicit distance metric. As such, autoencoders could be used to learn a distance metric between two samples from the input space, unlocking the application of semisupervised algorithms with stricter prerequisites. [21] Grandini, Margherita, Enrico Bagli, and Giorgio Visani. "Metrics for multi-class classification: an overview." arXiv preprint arXiv:2008.05756 (2020). [22] Zhang, Zhao, Tommy WS Chow, and Mingbo Zhao. "M-Isomap: Orthogonal constrained marginal isomap for nonlinear dimensionality reduction." IEEE transactions on cybernetics 43.1 (2012): 180-191. [23] Nath, Vishwesh, et al. "Warm start active learning with proxy labels and selection via semisupervised fine-tuning." International conference on medical image computing and computer-assisted intervention. Cham: Springer Nature Switzerland, 2022. [24] Zgurovsky, M., Sineglazov, V., Chumachenko, E. (2021). Classification and Analysis of Multicriteria Optimization Methods. In: Artificial Intelligence Systems Based on Hybrid Neural Networks. Studies in Computational Intelligence, vol 904. Springer, Cham. https://doi.org/10.1007/978-3-030-48453-8_ [25] V.M. Sineglazov, K.D. Riazanovskiy, O.I. Chumachenko, (2020). Multicriteria conditional optimization based on genetic algorithms. In: System research and information technologies, No. 3 (2020) . DOI: https://doi.org/10.20535/SRIT.2308-8893.2020.3.07 [26] Sineglazov , V., & Kot, A. (2021). Design of hybrid neural networks of the ensemble structure. Eastern-European Journal of Enterprise Technologies, 1(4 (109), 31–45. https://doi.org/10.15587/1729-4061.2021.225301 [27] Zgurovsky, M., Sineglazov, V., Chumachenko, E. (2021). Formation of Hybrid Artificial Neural Networks Topologies. In: Artificial Intelligence Systems Based on Hybrid Neural Networks. Studies in Computational Intelligence, vol 904. Springer, Cham. https://doi.org/10.1007/978-3-030-48453-8_3 [28] V. Sineglazov, Main features of terrain-aided navigation systems, 2014 IEEE 3rd International Conference on Methods and Systems of Navigation and Motion Control (MSNMC), Kiev, Ukraine, 2014, pp. 49-52, doi: 10.1109/MSNMC.2014.6979728.