Hyperspectral images (HSIs) have a wide field of view and rich spectral information, where each pixel represents a small area of the earth's surface. The pixel-level classification task of HSI has become one of the research hotspots in hyperspectral image processing and analysis. More and more deep learning methods have been proposed in recent years, among which convolutional neural network (CNN) is the most influential. However, it is dificult for CNN-based models to obtain the global receptive field in HSI classification task. Besides, most of the self-supervised training methods are based on sample reconstruction, and it is not easy to achieve efective use of unlabeled samples. In this paper, we propose a novel convolutional embedding module, combined with the Transformer blocks, which successfully improves the context-awareness while retaining the local feature extraction capability. Moreover, a new self-supervised task is designed to make more eficient use of unlabeled data. Our proposed pre-training task only masks the central token and reconstructs the central pixel from a learnable vector. It allows the model to capture the patterns between the central object and surrounding objects without labels.
[6], to classify the ground objects through spectral information. However, the imaging distance of HSI is far Hyperspectral images are generally composed of dozens away, and there are many interference factors in this proto hundreds of bands and have the characteristics of low cess, so that the spectral curve of diferent surface objects spatial resolution and high spectral resolution. The spec- is not always easy to distinguish. This creates dificultral information provides the possibility to distinguish the ties for these methods to achieve good performance in corresponding land covers, which has spawned various complex scenes. In recent years, deep learning methods research fields. Among them, pixel-level hyperspectral have gradually become popular, in which CNN-based image classification is the most concerned one in the methods are dominant. Hu et al. [7] made a preliminary community. Its main task is to assign a class label to attempt that several 1-D convolutional layers are stacked each pixel, somewhat like semantic segmentation in the to extract local spectral information, and many classical computer vision (CV) field. Diferent from RGB image, data augmentation methods in CV have been introduced. hyperspectral image is high-dimensional data. In order to Roy et al. [8] combined 3D-CNN and 2D-CNN to achieve avoid the curse of dimensionality, principal component hierarchical feature learning. In addition, other neural analysis (PCA) [1] and independent component analysis networks have also achieved good performance. Zhou [2] are widely used for redundancy elimination. et al. [9] designed a two-branch Long Short-Term Mem
So far, many hyperspectral image classification meth- ory network (LSTM) to extract spectral information and ods have been proposed, but deep learning methods have spatial information respectively. He et al. [10] proposed taken the lead. According to the diferent techniques a pure multilayer perceptron (MLP) network, proving used, it can be divided into traditional methods and deep that the MLP network still has potential. Hong et al. learning-based methods. In early research, people mostly [11] designed a mini-batch graph neural network. It selected a single pixel and all its spectral information is worth mentioning that the recently prevalent Transas the training sample and rely on the traditional clas- former model has also been introduced. Hu et al. [12] sifiers, such as logistic regression [ 3], decision tree [4], used 1-D convolution as an embedding layer combined random forest [5], and support vector machine (SVM) with Transformer Block. Hong et al. [13] analyzed the difference between Transformer and other classical neural CDCEO 2022: 2nd Workshop on Complex Data Challenges in Earth networks in detail and proposed a ViT-based Spectral*OCboserrrveasptioonn,dJinuglya2u5t,h2o0r2.2, Vienna, Austria Former for spectral information learning. Zhong et al. $ 2070276050@email.szu.edu.cn (Y. Wang); shuguoj@foxmail.com [14] proposed a spatial–spectral Transformer network (S. Jiang); m.xu@szu.edu.cn (M. Xu); shuyu-zhang@szu.edu.cn and a model structure search framework. Dang et al. [15] (S. Zhang); senjia@szu.edu.cn (S. Jia) combined spectral-spatial attention module with densely © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License connected Transformer blocks. Besides, self-attention CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) network is also used to address adversarial attacks that may be encountered in hyperspectral classification tasks [16].
However, limited by the size of the receptive field, it is dificult for CNN models to capture the global relationship. Meanwhile, deep learning models are datadriven which means that more labeled data leads to better model performance. But, obtaining such a large number of labeled samples in practical applications is expensive.
How to efectively use unlabeled data has become an urgent need. The self-supervised pre-training method in HSI classification task is still stuck in the autoencoderbased sample reconstruction [17]. This article proposes a band-grouping-based 3D convolutional Transformer (BG3DCT) and a new self-supervised task for model pretraining. The main contributions are listed as follows: • A novel band-grouping-based 3D convolutional
Transformer is designed for HSI classification.
We replace the commonly used linear embedding module with a well-designed 3D convolutional embedding module, combined with the spectral segmentation strategy, to achieve eficient spatialspectral feature embedding in each sub-band. • According to the characteristics of hyperspectral data, a new pre-training task is proposed. In the process of masking and reconstructing the center pixel, the model’s ability to capture the relationship between the center pixel and surrounding pixels is improved. Compared to the overall sample reconstruction task, center-masked pretraining task is more eficient for the representation of center area in the pre-training stage. • A series of comparative experiments and ablation experiments demonstrate the efectiveness of our proposed pre-training method and BG3DCT network. In particular, our proposed pre-training method can alleviate the instability of results caused by random sampling in the limited training samples scenario.
HSIs, the ViT network is not well compatible with hyperspectral image. When the training sample is limited, the linear embedding module cannot suficiently characterize the spatial-spectral features. Meanwhile, CNN-based module is more adaptable to this situation, while also being able to capture the local texture information. So we design a band-grouping-based 3D convolutional embedding module for HSI embedding. Firstly, we perform PCA processing on the input samples and employ a spectral partition strategy to divide the spectra into several sub-bands of equal length. Because the spectral curves of objects often have local diferences, the extraction of 3DCNN on sub-band is more eficient than full-band. Then, paralleled 3D convolution extraction are performed on each sub-band twice, and the 3D batch normalization operation is followed to unify the feature deviations generated from each sub-band. Finally, we concatenate the features to maintain the relative positional relationship between sub-bands and a lightweight 2D-CNN is used for feature fusion and compression. In particular, paralleled 3D convolution operation have a simple implementation, and the convolution function includes a grouped convolution option. The detailed description of BG3DCE module is listed in table 1. 2.1.2. Transformer Encoder
proposed method will be introduced in detail in section II. The descriptions of the comparative experiments and result analysis will be provided in section III, and section IV presents the conclusions.
the model go deeper, but HSI data is limited, so it is dififcult for us to stack modules as simply as the model in 2. Methodology CV task. In contrast, The multi-head attention module can make up for the shortcomings of CNN here and ef2.1. BG3DCT Network fectively model the relationship between ground objects. So, the combination of CNN and Transformer is compleThe BG3DCT network has three parts, the band- mentary and powerful. A standard Transformer mainly grouping-based 3D convolutional embedding (BG3DCE) comprises position encoding, multi-head attention, and module, Transformer encoder, and MLP head. The spe- feedforward layers. Since the convolution features concific design is as follows. tains position information, the positional encoding is not used here, and the embedded spatial-spectral features are directly input into the Transformer blocks. Finally, we add an average pooling layer to achieve the global representation and get the classification results through a MLP layer. 2.2. Center-mask Pre-training Task Today, most hyperspectral image classification methods are patch-based. The model’s input not only is the spectral curve of the centre pixel but also contains its neighbour region, which is generally a square area and makes the input more distinctive. Inspired by the form of the training sample, we propose the center-masked pre-training task, which is similar to MAE [18] but our method is easier to implement.
The flowchart of our proposed pre-training task is shown in Fig. 2. The Encoder is a BG3DCT network but removes the average pooling layer and MLP layer.
The decoder consists of two layers of standard Transformer encoders, which are only used in the pre-training stage. Given an input sample X and center pixel vector , the latent representation of the input sample is E (embedded by the BG3DCE module). Unlike self-supervised pre-training in the CV field, RGB images cannot directly ifnd areas that need to be focused on, but the neighborhood areas of HSI samples serve for the central pixel. So, our masking target can select the essential part in the training sample, namely the center pixel. Therefore, we replace the token in the middle of the sequence E with a learnable vector. Then, the masked sequence is input into the decoder, and the pixel-level reconstruction is performed by a MLP head to obtain the reconstruction result ˆ of the center pixel. The target of the center-masked pre-training task is to reconstruct the centre pixel as eficiently as possible so that the encoder can better learn the relationship between the centre pixel and the neighbour pixels without labels. The reconstruction target can be formulated as: (, ˆ) = min | − ˆ|2
(1) where is the similarity function. In the deep learning framework, function is equivalent to the mean squared error (MSE) loss function.
partition of the training set and testing set are shown in the table 2 and table 3, respectively. We use three 180 bands after removing noise bands. The surface obmetrics to evaluate the classification results, overall accu- jects are mainly wetland vegetation, there are 20 kinds racy (OA), classwise average accuracy (AA), and kappa of objects, and the total number of labeled samples is coeficient ( ). All the experiments are conducted on a 77,937. computer with an Intel Xeon Platinum 8260 CPU, 64-GB RAM and an NVIDIA Tesla P100-16GB GPU. The model 3.2. Comparative Experiment structure and parameter settings of the comparison methods comply with open source codes or corresponding pa- To demonstrate the superiority of our proposed method, pers. For our proposed model, the patchsize is set to 13, we select five state-of-the-art methods on two public the spectral dimension is 80 after PCA, and the number datasets, Salinas and YRE, including four CNN-based of sub-bands is 10. The embedding size of each token is methods and one classical Transformer network. They set to 128. The learning rate is set to 0.001, and Adam are CNNHSI [19], FC3D [20], HybridSN [21], TwoCNN is adopt as the gradient descent optimizer. Meanwhile, [22], and Vision Transformer (ViT) [23]. Among them, all the experiments are repeated ten times to smooth out several methods based on 2D-CNN are distinguished in errors caused by random sampling. The setting of the the size of the convolution kernel and the structure decenter-masked pre-training is the same. sign. CNNHSI stacks several 2-D Convolution layers with 1×1 kernel size. TwoCNN is a dual-branch CNN with a 2D-CNN and a 1D-CNN to extract spatial information 3.1. Datasets Description and spectral information, respectively. FC3D is a pure 3D3.1.1. Salinas Dataset CNN network, and HybridSN uses 3D convolution and 2D convolution successively for hierarchical feature exThe salinas dataset, collected by the AVIRS sensor in the traction. ViT divides the input samples into equal-sized Salinas Valley, USA, has an image size of 512 × 217 and a patches, obtains the embedded tokens through linear spatial resolution of 3.7 meters. After noise band removal, embedding module, and then inputs them into the Trans204 bands are remained. There are 16 kinds of ground former encoder. objects in the dataset, with 56,975 samples that can be The results of the comparative experiments are shown used for pixel-level classification. in table 4 and table 5. Our method has obtained obvious advantages and achieved the best or second-best results 3.1.2. YRE Dataset in each class, reflecting our approach’s superiority and robustness. Under the setting of training with limited samples, CNNHSI achieves excellent classification results due to its lightweight network structure. Limited by the large model size, HybridSN, FC3D, and TwoCNN fail to
Classification Accuracy (%) and Kappa Measure for The Salinas Ablation Study Results Toward The Center-Masked PreClassification Accuracy (%) and Kappa Measure for The YRE BG3DCT module. The results are shown in table 6 that Class
ViT HybridSN FC3D CNNHSI TwoCNN Ours ViT HybridSN FC3D CNNHSI TwoCNN Ours
In this article, we creatively propose a band-groupingbased convolutional embedding module to extract spatialspectral information in each sub-bands. The Transformer module is used to model the global relationship between surface objects. Additionally, for efective use of unlabeled data, we design a new unsupervised pre-training task for hyperspectral classification. Through the mask and reconstruction process of the token generated from the central area, the model can initialize the backbone network without labeled data and provide a more stable model performance. To fully evaluate our proposed methods, we conducted a series of comparative experiments and ablation experiments on two public datasets, Salinas and YRE. The experimental results prove the efectiveness and superiority of our method.
ral Science Foundation of China under Grant 41971300, and Grant 61901278; in part by the Key Project of
der Grant 2020ZDZX3045; in part by the Guangdong
Grant 2022A1515011290; in part by the Natural Sci- [12] X. Hu, W. Yang, H. Wen, Y. Liu, Y. Peng, A ence Foundation of Guangdong Province under Grant lightweight 1-d convolution augmented trans2021A1515011413; in part by Shenzhen Scientific Re- former with metric learning for hyperspectral imsearch and Development Funding Program under Grant age classification, Sensors 21 (2021) 1751. 20200803152531004. [13] D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, J. Chanussot, Spectralformer: Rethinking hyperspectral image classification with transformReferences ers, IEEE Transactions on Geoscience and Remote [1] M. D. Farrell, R. M. Mersereau, On the impact of pca Sensing 60 (2022) 1–15. doi:10.1109/TGRS.2021. dimension reduction for hyperspectral detection [14] 3Z1.Z30h7o1n6g,. Y. Li, L. Ma, J. Li, W.-S. Zheng, Spectralof dificult targets, IEEE Geoscience and Remote spatial transformer network for hyperspectral imSensing Letters 2 (2005) 192–195. age classification: A factorized architecture search [2] S. Moussaoui, H. Hauksdottir, F. Schmidt, C. Jutten, framework, IEEE Transactions on Geoscience and J. Chanussot, D. Brie, S. Douté, J. A. Benediktsson, Remote Sensing (2021).
On the decomposition of mars hyperspectral data [15] L. Dang, L. Weng, W. Dong, S. Li, Y. Hou, Spectralby ica and bayesian positive source separation, Neu- spatial attention transformer with dense connection rocomputing 71 (2008) 2194–2208. for hyperspectral image classification, Computa[3] Y. Qian, M. Ye, J. Zhou, Hyperspectral image clas- tional Intelligence and Neuroscience 2022 (2022). sification based on structured sparse logistic re- [16] Y. Xu, B. Du, L. Zhang, Self-attention context netgression and three-dimensional wavelet texture fea- work: Addressing the threat of adversarial attacks tures, IEEE Transactions on Geoscience and Remote for hyperspectral image classification, IEEE TransSensing 51 (2012) 2276–2291. actions on Image Processing 30 (2021) 8671–8685. [4] S. Kuching, The performance of maximum likelihood, spectral angle mapper, neural network and [17] dYo.iC:1h0e.n1,1Z0.9L/inT,IXP..2Z0h2a1o.,3G1.1W89an7g7,. Y. Gu, Deep decision tree classifiers in hyperspectral image anal- learning-based classification of hyperspectral data, ysis, Journal of Computer Science 3 (2007) 419–423. IEEE Journal of Selected topics in applied earth ob[5] J. Xia, P. Ghamisi, N. Yokoya, A. Iwasaki, Random servations and remote sensing 7 (2014) 2094–2107. forest ensembles and extended multiextinction pro- [18] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, ifles for hyperspectral image classification, IEEE Masked autoencoders are scalable vision learners, Transactions on Geoscience and Remote Sensing arXiv preprint arXiv:2111.06377 (2021). 56 (2017) 202–216. [19] S. Yu, S. Jia, C. Xu, Convolutional neural networks [6] M. Chi, R. Feng, L. Bruzzone, Classification of hy- for hyperspectral image classification, Neurocomperspectral remote-sensing data with primal svm puting 219 (2017) 88–98. for small-sized training dataset problem, Advances [20] M. Ahmad, A. M. Khan, M. Mazzara, S. Distefano, in space research 41 (2008) 1793–1799. M. Ali, M. S. Sarfraz, A fast and compact 3-d cnn [7] W. Hu, Y. Huang, L. Wei, F. Zhang, H. Li, Deep for hyperspectral image classification, IEEE Geoconvolutional neural networks for hyperspectral science and Remote Sensing Letters (2020). image classification, Journal of Sensors 2015 (2015). [21] S. K. Roy, G. Krishna, S. R. Dubey, B. B. Chaudhuri, [8] S. K. Roy, G. Krishna, S. R. Dubey, B. B. Chaudhuri, Hybridsn: Exploring 3-d–2-d cnn feature hierarHybridsn: Exploring 3-d–2-d cnn feature hierar- chy for hyperspectral image classification, IEEE chy for hyperspectral image classification, IEEE Geoscience and Remote Sensing Letters 17 (2020) Geoscience and Remote Sensing Letters 17 (2019) 277–281.
277–281. [22] J. Yang, Y.-Q. Zhao, J. C.-W. Chan, Learning and [9] F. Zhou, R. Hang, Q. Liu, X. Yuan, Hyperspectral im- transferring deep joint spectral–spatial features for age classification using spectral-spatial lstms, Neu- hyperspectral classification, IEEE Transactions on rocomputing 328 (2019) 39–47. Geoscience and Remote Sensing 55 (2017) 4729– [10] X. He, Y. Chen, Modifications of the multi-layer 4742.
perceptron for hyperspectral image classification, [23] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. WeisRemote Sensing 13 (2021) 3547. senborn, X. Zhai, T. Unterthiner, M. Dehghani, [11] D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, M. Minderer, G. Heigold, S. Gelly, et al., An image is J. Chanussot, Graph convolutional networks for worth 16x16 words: Transformers for image recoghyperspectral image classification, IEEE Transac- nition at scale, arXiv preprint arXiv:2010.11929 tions on Geoscience and Remote Sensing 59 (2020) (2020). 5966–5978.