=Paper=
{{Paper
|id=Vol-2665/paper8
|storemode=property
|title=Hyperspectral data dimensionality reduction using nonlinear autoencoders 
|pdfUrl=https://ceur-ws.org/Vol-2665/paper8.pdf
|volume=Vol-2665
|authors=Evgeny Myasnikov
}}
==Hyperspectral data dimensionality reduction using nonlinear autoencoders ==
<pdf width="1500px">https://ceur-ws.org/Vol-2665/paper8.pdf</pdf>
<pre>
   Hyperspectral data dimensionality reduction using
               nonlinear autoencoders
                                                               Evgeny Myasnikov
                                            Geoinformatics and Information Security department
                                                    Samara National Research University;
                       Image Processing Systems Institute of RAS - Branch of the FSRC "Crystallography and Photonics" RAS
                                                                 Samara, Russia
                                                              mevg@geosamara.ru

    Abstract—The known feature of hyperspectral images is a                  allowed to outperform the PCA technique both in terms of
high spectral resolution, which allows us to identify materials              the reconstruction error and classification accuracy.
and classify objects in images with high accuracy. However
hyperspectral images contain substantial redundancy, which                      However, it was also shown [7,8] that the nonlinear
can be eliminated with the aid of dimensionality reduction                   mapping technique [9] have advantages over the PCA in
techniques. In this paper, we propose and study several                      terms of classification and segmentation quality of
dimensionality reduction techniques based on the pretraining                 hyperspectral images. For this reason, in this paper, we
the encoder-decoder neural network with the results of the                   study the possibility to train the autoencoder–like
nonlinear mapping and principal component analysis
techniques. The experiments performed on an open dataset                     architecture to capture the nonlinear mapping. In particular,
show that the proposed techniques both provide the                           we split the autoencoder into encoder and decoder and train
discriminative low-dimensional features and allow us to                      both parts separately using the results of nonlinear mapping
reconstruct source hyperspectral data with little error.                     and investigate the effect of the subsequent fine-tuning of
  Keywords—autoencoder, hyperspectral images, nonlinear                      the whole network.
mapping, principal component analysis                                            The structure of the paper is as follows. In the next
                                                                             Section II, we give necessary theoretical information on the
                         I.    INTRODUCTION                                  neural network architecture and the nonlinear mapping
    Hyperspectral images are widely used nowadays in                         algorithm. In Section III we describe the training procedures
different fields such as agriculture, medicine, biology,                     used in the experimental study and describe the results of
chemistry, and so on. The known feature of hyperspectral                     experiments. The conclusions and the list of references are
images is high spectral resolution, which allows us to                       given at the end of the paper.
identify materials and classify depicted images with high                                            II.   METHOD
accuracy.
                                                                             A. Autoencoder Neural Network
    However hyperspectral images contain substantial
redundancy, which can be eliminated with the aid of                              The autoencoder neural network proposed in [5] was
dimensionality reduction techniques. The images obtained                     earlier referred to as the autoassociative neural network. It
after the dimensionality reduction stage can be processed                    consists of two consecutive parts called the encoder and
efficiently as much less data volume is involved in                          decoder.
processing. It is worth noting that dimensionality reduction                     The encoder part takes a multidimensional vector x ϵ RM
techniques are often used in different problems of image                     as input and produces corresponding low-dimensional
analysis (see [1-3], for example). The key requirement to the                representations y ϵ Rm so that m<M. The encoder consists of
dimensionality reduction procedures is the possibility to                    at least two fully – connected layers. The first layer contains
preserve the quality of the solution of applied problems that                some number of neurons (defined by the parameters of the
is classification, segmentation, material detection, and so on.              neural network architecture) connected to all the components
    The most commonly used techniques for the                                of an input vector. The last layer of the encoder contains the
dimensionality reduction of hyperspectral data are linear                    number of neurons equal to the desired dimensionality of the
techniques such as Principal Component Analysis (PCA).                       reduced space.
While     a    number       of general-purpose       nonlinear                   The decoder usually has the mirror-reflected architecture.
dimensionality reduction procedures exist [4], their use in                  It has the same number of layers with the same number of
hyperspectral image analysis is limited as many of them do                   neurons, but this is not the necessary requirement. Anyway,
not provide the ability to restore source hyperspectral data as              the input layer of the decoder takes the reduced
such procedures provide only one-way data mapping.                           representation y ϵ Rm from the output of the encoder and
    In the last years, neural network approaches become                      restores the multidimensional vectors ~x ϵ RM. So the output
more and popular. In particular, autoencoder neural networks                 layer of the decoder has the number of neurons equal to the
[5] were used for the dimensionality reduction of                            input dimensionality M. The number of hidden layers and
hyperspectral images. Such neural networks perform both                      neurons is defined by the parameters of the neural network
nonlinear dimensionality reduction and provide the inverse                   architecture.
mapping, which allows us to restore the source hyperspectral
                                                                                 As the number of neurons in the output layer of the
data up to some reconstruction error.
                                                                             encoder is less than the number of neurons in the input and
    Recently, it was shown [6] that the autoencoder network                  hidden layers, this layer is often referred to as a bottleneck
can be pretrained using principal component analysis                         layer, and the whole network architecture is often referred to
technique, and its use for the dimensionality reduction                      as a bottleneck architecture.


Copyright © 2020 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Image Processing and Earth Remote Sensing

    The autoencoder architecture is usually trained in self-                                            nonlinear mapping consists of the initialization of the
learning mode by applying the same multidimensional                                                     coordinates yi with the results of the principal component
vectors x ϵ RM to both input and output layers of the                                                   analysis with the subsequent refinement of yi using the
autoencoder. The training process itself is based on the                                                stochastic gradient descent. The optimization process
minimization of the following cast function:                                                            (refinement) stops when the coordinates of the data points yi
                                                                                                        in the reduced space become stable.
                                                 N
                                          1
                                                 x x                                         
                                                                    ~       2
                              E                            i           i
                                                                                                        C. The methods used in the study
                                          N
                                                 i 1                                                       As it was outlined in the introduction, in this paper, we
where N is the number of samples, and x ϵ RM, ~x ϵ RM are                                               study several variants of training the autoencoder-like
inputs and outputs of the network. After training the encoder                                           encoder-decoder network. In particular, we consider the
can be used to perform the dimensionality reduction of the                                              following techniques:
source data (direct mapping), and decoder can be used to
restore the source data by its reduced representation (inverse                                              - The autoencoder network pretrained with the results of
mapping).                                                                                               the PCA technique (AE-PCA), as it is described in [6];

    In this paper, we study, if the encoder and decoder parts                                             - The neural network with encoder and decoder (ED-
can be trained separately to force the neural network to                                                NLM) trained separately using the results of the nonlinear
perform the mapping with the desired properties. It was                                                 mapping technique;
shown earlier that the separate pre-training of encoder and                                                 - The same autoencoder network pretrained with the
decoder with the PCA results helped to perform the training                                             results of the nonlinear mapping technique and fine-tuned
more efficiently compared to the standard training.                                                     using the standard approach (AE-NLM).
    In particular, the approach proposed in [6] consists of the                                                              III.    EXPERIMENTS
following steps: perform the PCA for the input dataset; pre-
train the encoder to produce the PCA results for the input                                                  In this section, we describe the results of the experiments,
data; pre-train the decoder to produce the input data for the                                           which were performed using the Indian Pines dataset. This
encoded data; fine-tune the whole network according to the                                              dataset was acquired using the AVIRIS hyperspectral sensor.
standard scheme.                                                                                        This dataset contains 145 x 145 image pixels and 224
                                                                                                        spectral components [10]. Due to the high noise and water
    In this paper, we follow the similar scheme but use the                                             absorption in the source image, we used the version
results of the nonlinear mapping algorithm instead of the                                               containing 204 spectral channels.
PCA, and perform the fine-tuning optionally to study if such
an approach can be more efficient than the standard PCA,                                                    In all the described experiments, for the implementation
nonlinear mapping or the proposed recently autoencoder                                                  of the neural networks, we used the Keras framework and
pretrained with the PCA [6].                                                                            Python language. The experiments were carried out on
                                                                                                        GeForce GTX 1070 ti.
B. Nonlinear Mapping
                                                                                                            For each considered neural network technique, we varied
    The nonlinear mapping is a numerical procedure that
                                                                                                        the number of hidden layers in the encoder and decoder and
performs the mapping (nonfunctional) of data into low-
                                                                                                        performed experiments for one and two hidden layers that
dimensional space so that the data structure is preserved (see
                                                                                                        correspond to four and six layers in the corresponding
[8] for example). This structure is defined in nonlinear
                                                                                                        autoencoder networks.
mapping by all the pairwise distances between the points in
the dataset. The Euclidean distance d() is usually used to                                                  The number of neurons in the input layer of the encoder
measure the distances.                                                                                  and the output layer of the decoder was defined by the
                                                                                                        dimensionality of the input space that is the number of
    As the pairwise distances cannot be preserved exactly in
                                                                                                        channels in the hyperspectral image. The number of neurons
a common case, the so-called data mapping error is
                                                                                                        in the bottleneck layer varied from 1 to 10 according to the
introduced:
                                                                                                        dimensionality of the reduced space. We also varied the
                             N                                                                          number of neurons in the hidden layers. In particular, we
                            d ( x , x )  d ( y y )                                      
                                                                                             2
                                      i, j          i       j                   i,   j
                                                                                                        used 64, 128, and 256 neurons in hidden layers.
                       i, j 1 (i  j )
                                                                                                            According to the recommendations given in [6], we used
Here N is the number of data points, d(xi,xj) is the distance                                           ReLU activation functions for hidden layers and linear
between points xi and xj in the multidimensional space,                                                 activations in the output layers of the encoder and decoder.
d(yi,yj) is the distance between the corresponding points yi, yj                                        Analogously, we used Adam optimizer [11] with the default
in the reduced space, µ and  are some constants. Usually, µ                                            parameters. The batch size was set to 16, however, we
is the inversion of the sum of square distances between all                                             suppose that a bigger batch size could also be used.
the possible pairs of data points in multidimensional space,
and ij are equal to one.                                                                                   To measure the effectiveness of each particular approach,
                                                                                                        we estimated both the reconstruction error as it is defined in
    The minimization of the data mapping error is usually                                               (1) and the classification accuracy using the reduced
performed using the gradient descent technique. The                                                     representation. The latter indicator plays an important role in
coordinates of data points yi ϵ Rm are the tunable parameters.                                          hyperspectral image analysis problems, for example, in
                                                                                                        vegetation type recognition [12].
   In this paper, we use the stochastic gradient descent
based on mini-batches to minimize the data mapping error.                                                  For the latter indicator, we used the overall accuracy of
The overall algorithm for dimensionality reduction using the                                            the one nearest neighbor (1-NN) classifier. The accuracy



VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020)                                                                               34
Image Processing and Earth Remote Sensing

itself was measured as a fraction of correctly classified image           quality indicator. The experiment was carried out for a
pixels. To measure the accuracy, at first, we performed                   different number of layers and neurons.
dimensionality reduction using one of the studied techniques
for all the pixels in the considered image. Then we split all                 As can be seen in the figure, the reconstruction error
the ground truth pixels into training and testing sets in the             decreases with the growth of the dimensionality m of the
proportion 60/40. After that, we trained the classifier using             reduced space defined by the number of neurons in the
the training set and estimated its accuracy using the test one.           bottleneck layer, which is an expected result.

    In our first experiment, we compared different techniques                 While we cannot highlight any winner technique in this
described in Subsection II.C and different architectures from             experiment, we should note, that the AE-NLM technique
the viewpoint of the reconstruction error (1). The results of             often shows better results. It means that the nonlinear
this experiment are shown in Fig. 1. In particular, we pre-               mapping result, which was used for training, provide the
trained the encoder and decoder of the AE-PCA network for                 ability to restore the source data with quite a good quality.
50 iterations, fine-tuned the entire network for 50 iterations,           This also means that the decoder trained on the NLM data
and then measures the reconstruction quality.                             can be used as an inverse mapping for the NLM.




                                   (a)                                                                        (a)




                                  (b)                                                                         (b)




                                                                                                              (c)
                                   (c)
                                                                          Fig. 2. The classification accuracy for different techniques and network
Fig. 1. The reconstruction error for different techniques and network
                                                                          architectures (a-c).
architectures (a-c).
                                                                              In our second experiment, we compared the considered
    For the AE-NLM network, we trained the network with
                                                                          techniques from the viewpoint of the classification accuracy.
the same strategy, but used the NLM results instead of the
                                                                          The results of this experiment are shown in Fig. 2. In this
PCA results at the pretraining stage. For the ED-NLM
                                                                          figure, we added the results for the classical linear (PCA) and
network, we trained separately encoder and decoder for 100
                                                                          nonlinear (NLM) dimensionality reduction techniques.
epochs. After the training, we measured the error (1) as the



VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020)                                                         35
Image Processing and Earth Remote Sensing

    As can be seen, the proposed techniques provided better                  [2]  M.V. Gashnikov, “Optimization of the multidimensional signal
results than the classical approaches in most cases. Again, it                    interpolator in a lower dimensional space,” Computer Optics, vol. 43,
                                                                                  no. 4, pp. 653-660, 2019. DOI: 10.18287/2412-6179-2019-43-4-653-
is difficult to outline any approach. Nevertheless, we do not                     660.
observe any substantial advantages in the fine-tuning of the                 [3] E.V. Myasnikov, “The study of dimensionality reduction methods in
NLM initialized network over the version with separate                            the task of browsing of digital image collections,” Computer Optics,
encoder and decoder.                                                              vol. 32, no. 3, pp. 296-301, 2008.
                                                                             [4] J.A. Lee and M. Verleysen, “Nonlinear Dimensionality Reduction,"
                      IV.       CONCLUSION                                        Springer, 2007.
    In this paper, we studied several dimensionality reduction               [5] M.A. Kramer, “Nonlinear principal component analysis using
neural network techniques based on autoencoder                                    autoassociative neural networks,” AIChE J., vol. 37, pp. 233-243,
                                                                                  1991.
architecture. We compared the proposed techniques from the
                                                                             [6] E. Myasnikov, “Dimensionality Reduction of Hyperspectral Images
viewpoint of the reconstruction error and the accuracy of the                     using Autoassociative Neural Networks,” IEEE Proc. of International
per-pixel classification.                                                         Multi-Conference on Engineering, Computer and Information
                                                                                  Sciences (SIBIRCON), pp. 0591-0595, 2019.
    We showed that the proposed techniques outperformed
                                                                             [7] E. Myasnikov, “Evaluation of nonlinear dimensionality reduction
the baseline (PCA and NLM) approached in terms of the                             techniques for classification of hyperspectral images,” CEUR
classification accuracy in almost all the considered cases.                       Workshop Proceedings, vol. 2268, pp. 147-154, 2018.
The decoder trained using the results of the NLM can be                      [8] S. A. Bibikov, N. L. Kazanskiy and V. A. Fursov, “Vegetation type
successfully used as an inverse mapping for hyperspectral                         recognition in hyperspectral images using a conjugacy indicator,”
image analysis.                                                                   Computer Optics, vol. 42, no. 5, pp. 846-854, 2018. DOI:
                                                                                  10.18287/2412-6179-2018-42-5-846-854.
                        ACKNOWLEDGMENT                                       [9] J.W. Sammon, “A nonlinear mapping for data structure analysis,”
                                                                                  IEEE Transactions on Computers, vol. 18, no. 5, pp. 401-409, 1969.
    The work was partly funded by RFBR according to the                      [10] M.F. Baumgardner, L.L. Biehl and D.A. Landgrebe, “220 Band
research project 18-07-01312-a in parts of «2. Method» - «3.                      AVIRIS Hyperspectral Image Data Set: June 12, 1992 Indian Pine
Experiments» and by the Russian Federation Ministry of                            Test Site 3,” Purdue University Research Repository, 2015.
Science and Higher Education within a state contract with                         DOI:10.4231/R7RX991C.
the «Crystallography and Photonics» Research Center of the                   [11] D. Kingma and J. Ba, “Adam: A Method for Stochastic
RAS in parts «1. Introduction» and «4. Conclusion».                               Optimization,” arXiv: 1412.6980v8, 2017.
                                                                             [12] S.A. Bibikov, N.L. Kazanskiy and V.A. Fursov, “Vegetation type
                             REFERENCES                                           recognition in hyperspectral images using a conjugacy indicator,”
                                                                                  Computer Optics, vol. 42, no. 5, pp. 846-854, 2018. DOI:
[1]   E.A. Dmitriev and V.V. Myasnikov, “Comparative study of                     10.18287/2412-6179-2018-42-5-846-854.
      description algorithms for complex-valued gradient fields of digital
      images using linear dimensionality reduction methods,” Computer                                            .
      Optics, vol. 42, no. 5, pp. 822-828, 2018. DOI: 10.18287/2412-6179-
      2018-42-5-822-828.




VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020)                                                              36

</pre>