Hyperspectral images have shown great potential for the target detection task. These images collect the reflectance physical value over a large electromagnetic spectrum providing a fingerprint that characterizes uniquely distinct materials. In this work, a framework is developed to recognize diferent materials using several approaches ranging from classical methods to deep learning ones. Diferent learning paradigms are investigated considering both supervised and self-supervised methods. The main diference between these approaches concerns the labeling process. Indeed, while the former method requires labeling the data, the latter approach is based on pseudo-labels generation described in this contribution.
nology that allows for the collection of a wide range of spectral data acquired by remote sensors. It has been shown to be useful for various applications, including object detection, classification, and material recognition.
terial fingerprints that can be used to identify diferent materials.
In recent years, there has been an increasing interest in
developing machine learning models that can accurately
recognize materials from hyperspectral images. Deep
learning has emerged as a promising approach to solving
complex problems in various fields. Among the
diferent deep learning models, convolutional neural networks
(CNNs) [
classical and deep learning approaches for material recognition in hyperspectral images. Diferent learning paradigms, including supervised and self-supervised methods, are investigated and evaluated for their performance on a benchmark dataset. The approach demonnized by CINI, May 29–31, 2023, Pisa, Italy ∗Corresponding author.
HyperHound is a framework developed specifically for analyzing hyperspectral images. It has been designed to allow for easy implementation and testing of various models for target detection. This framework comes with a broad range of capabilities, with its main features described below and provided with a simple user interface (UI) shown in Fig. 1: • Compatibility with the PIX format: support loading files in PCI (Geomatics Database File) format, splitting them into smaller patches, and visualizing them for the analysis process. • Datasets: Integration of both publicly available and privately collected datasets to enable model evaluation and comprehensive data analysis. • Implementation of classic target detection models: target detection is a critical task in computer vision, which involves identifying specific objects of interest within an image, HyperHound implements several algorithms that provide a solid foundation for measuring the performance of newer and more advanced models. Implementing these classic models allows us to compare the results of diferent models and evaluate their relative strengths and weaknesses. Some of the classical models implemented are Euclidean distance, CEM, MF, and ACE. • Data labeling: the interface of HyperHound provides two options for labeling data, individual that assume a Gaussian distribution. However, real-world hyperspectral data obtained through remote sensing often exhibits strong non-linearity and non-Gaussianity, which can result in a decline in the performance of these classical detection algorithms.
Self-supervised learning is a type of machine-learning technique in which a model is trained to learn patterns and relationships within a dataset without the need for explicit labeling or supervision. For the scope of this work, this method is used to learn a space topology to cluster similar hyperspectral signatures. In this sense, aFnigduarnea1ly: zUinIgofaHspyepcetrrhaolusingdnafrtaumreeowfotrhkelosaeldeicntgedSapliixneals. data starting from a reference signature, this algorithm can detect similar targets from the images analyzed. To overcome the labeling burden, an unsupervised method is used to generate pseudo-labels. The strategy used in this pixel labeling and bounding box selection. The work is described in the evaluation and results section labeled data can be used to train a classification and leverages a clustering pre-text task. Once pseudomodel. labels are generated, contrastive learning is used to train • Functionalities of Inference and Training: the model to cluster properly signatures belonging to HyperHound implements functionalities to both distinct classes. It is worth remembering that these are perform inference with pre- trained deep learn- the classes defined in a self-supervised manner, that is, ing models and training models on the fly from using an unsupervised pre-text task. A fully connected the interface. The inference process is optimized neural network was chosen to learn the distance metric by splitting the input image into smaller slices for class discrimination.
and processing them in parallel on a GPU.
• Database of spectral signatures: consisting of 3.3. Fully-connected neural network
laboratory- sampled materials collected from on- (FCNN)
line sources. This resource enables comparisons
between the reflectance of individual pixels and A fully-connected neural network consists of a series of
available materials, enabling the computation of fully connected layers that connect every neuron in one
similarity scores. layer to every neuron in the other layer.
• Atmospheric correction: Integration with Each neuron represents a computational unit that
proPy6S [
the Davinci-1 infrastructure, which comprises a total of 80 nodes, each equipped with four Nvidia A100 GPUs.
model using labeled training data, which consists of a set
of inputs and their corresponding outputs or class labels. 4. Evaluation and Results
The model’s parameters are updated iteratively during
the training phase to accurately predict the desired out- The following paragraphs begin by presenting one of the
puts. In the testing phase, the model is evaluated against datasets used to validate the self-supervised approach.
new input or test data to assess its ability to predict the Subsequently, the training and validation processes were
correct labels. With suficient training, the model can expanded to other hyperspectral datasets [
band AVIRIS sensor over Salinas Valley, California, and is characterized by high spatial resolution (3.7-meter pixels).
The area covered comprises 512 lines by 217 samples. 20 3.5. 3D Convolutional neural network water absorption bands were discarded: [108-112], [154167], for a total number of bands equal to 224. This image (3D-CNN) was available only as at-sensor radiance data. It includes Identifying ground objects in hyperspectral imaging re- vegetables, bare soils, and vineyard fields. Salinas groundquires both spectral and spatial information. To efec- truth contains 16 classes. tively classify these objects, a 3D convolutional neural network (CNN) was implemented. The network pro- 4.2. Data Labelling cesses each pixel of the images by considering the relation between adjacent channels, in addition to spatial patterns across neighboring pixels. The input of the 3D-CNN is a patch of 7x7 pixels, where N is the number of channels in the hyperspectral image. The architecture consists of a series of 3D convolutional layers, with decreasing iflter numbers leading to the last fully- connected layer.
This final layer takes the flattened concatenation of a set of feature maps from the last convolutional layers as input and outputs the probability of the center pixel of the input patch belonging to a target object. A scheme of this neural network is reported in Fig. 2.
erating samples that are labeled without full supervision. One method for accomplishing this is through the use of endmembers, which are defined as pure spectral signatures that can be linearly combined to represent the hyperspectral image pixels. Endmembers can be thought of as the basis vectors of a geometrical subspace. During image acquisition, due to the relatively low spatial resolution of hyperspectral sensors, some pixels may collect a mix of signatures from diferent materials. This means that each pixel can be seen as a superimposition of each endmember. By identifying the endmembers in the hyperspectral image, it is possible to obtain a set of pure spectral signatures that can be used to label the image data. Once the endmembers have been identified, they can be used in a variety of ways to label the image data. For example, one approach is to use spectral unmixing to estimate the abundance fractions of each endmember in each pixel. Nevertheless, some methods exist to unmix the pixel to find the basic constituents of each material, but their application doesn’t guarantee the optimality
the result of extensive hyperparameter optimization. To scale the search for optimizing hyperparameters, Ray Tune, a Python library designed to execute experiments and tune hyperparameters at any scale, was utilized. This was done using the Leonardo HPC system, specifically where are the randomly generated coeficients and represents the n endmembers extracted from the source image. This sampling is repeated to generate all the beling step, where the endmember corresponding to the highest coeficient is used to label the sample, in other words: (1) (2) dataset samples. The key step in this process is the la- other models on most of the tested datasets. = (
); .
datasets, each containing one or multiple classes to detect. For each dataset, one of the classes was designated as the target class, while the others were considered background classes.
The dataset used for the supervised task is a proprietary dataset. It consists of 4 images collected with a hyperspectral sensor during an aerial acquisition. The images were pre- processed by performing the L1 pre-processing chain, which consists of the following operations: • Spectral and Radiometric Calibration • Geo-Referencing • Geo-Rectification
These operations are missing the L2 pre-processing chain, which involves atmospheric correction and conversion of values to reflectance. In addition, the images have artifacts probably due to the vibrations the sensor was subjected to during the flight. With these limitations, a single pixel may contain a mixture of multiple hyperspectral signatures making the detection task harder.
each measuring 613 × 613 in size, for a total of 800 tiles.
Through ground surveys, it was determined that 18 of these tiles contained the targets to identify. Of these 18 tiles, 11 were included in the training-validation sets, while the remaining 7 tiles were used to test the models.
The labeling process is provided using the HyperHound interface. Through this interface, it is possible to display an image and collect a set of pixels to represent both target and background samples. This collection can be performed by using both bounding boxes or dot annotations, for a finer pixel selection. This procedure was repeated on all 18 tiles used for the training, validation, and test. A patch of dimension 7x7 was cropped around each pixel collected to provide the input in the form of images to the 3D CNN used for the training. The total number of patches collected for training and validation was nearly 1500 with a proportion of 1:14 between target and background samples. The partition of data into training, validation, and testing is summarized in Table 1. it is considered a true positive (TP). On the other hand, if there is no overlap between a predicted object and a ground-truth object, it is considered a false positive (FP). Finally, if a ground-truth label is not associated with any predicted object, the false negative (FN) count is in4.6. Performance creased by one unit. The results are provided in the table The objective was to identify target areas, and a detec- (2). tion metric was used to evaluate the model’s performance. The proposed model was evaluated on seven images The 1 score was chosen as the evaluation metric since it that were not included in the training or validation set handles class imbalances better than accuracy and other and achieved an 1 score of 0.6 in identifying the targets. metrics. An algorithm was developed to associate the An example of detection comparison on a test image model’s predictions with the ground-truth labels, and is reported in Fig. 5. The first patch (top left corner) assess the model’s performance. The output of the model represents the ground truth, that is, a completely black is a heat-map representing the probability of a pixel be- image with green boxes corresponding to the targets to longing to a target area. Therefore, the first step was to detect. The image is black in order to preserve sensitive apply a threshold to obtain a binary mask, where each information and the original image pertaining to this cluster of fully-connected pixels represents a predicted test is not shown. The remaining patches represent the object. Subsequently, if a predicted object partially or predictions of all the competing methods. Notably, only fully overlaps with an object in the ground-truth mask, the CNN3D was able to detect correctly all the targets on this test tile.
has been developed for hyperspectral image analysis. The framework provides an efective solution for analyzing hyperspectral data by applying deep learning techniques. Two types of deep learning models were analyzed using the framework: self-supervised and supervised. The selfsupervised approach is particularly useful in addressing the challenges of a lack of labeled data and the dificulty of pixel-level ground truth annotation. The model learns to predict features from the input data itself, without any explicit supervision. This approach is particularly efective when the ground truth data is not available, and it has shown good results in many literature datasets. However, the self- supervised models are less robust and their detection metrics are generally lower compared with supervised models. The supervised model, on the other hand, utilizes ground truth data to train the model. This type of model yields good results, providing accurate results even under diferent real-world conditions where classical and unsupervised models often fail.
In this study, it is highlighted that many hyperspectral datasets used as benchmarks lack suficient data, and the training and validation data are often highly correlated, resulting in models that are not robust to diferent realworld conditions. However, the supervised models have shown significant improvement and are particularly useful for man-in-the-loop applications. They provide an excellent tool for guiding and facilitating the task of an expert analyst in identifying targets, which is a challenging task in hyperspectral data analysis. Therefore, the HyperHound framework and supervised models provide a promising direction for hyperspectral data analysis, and they hold great potential for addressing the challenges of real-world applications.