Hyperspectral Denoising: From Conventional Techniques Towards Deep Learning Ones

Abstract

In this paper, we provide an overview on hyperspectral image denoising techniques. Additionally, we compare the conventional techniques with the deep learning ones applied to simulated and real datasets. The comparisons are performed in terms of signal to noise ratio and spectral angle distance for a simulated noisy dataset. Finally, we demonstrate the visual performance of the techniques applied to a real hyperspectral dataset. The results show that the established low-rank conventional techniques outperform the recent deep learning-based techniques used in the experiments.

1. Introduction

In the past decade, hyperspectral image (HSI) denoising has considerably evolved, from 2D and 3D full-rank conventional methods to low-rank ones. Recent advances in deep learning-based techniques have considerably inlfuenced the application of denoising in the computer vision community. However, HSIs have unique characteristics which distinguish them from the other digital images, e.g., RGB images. In this article, we provide an overview of the HSI denoising techniques. Additionally, we compare the conventional denoising techniques with the deep-learning ones.

2. Hyperspectral Denoising

An HSI can be modeled using

H = X + N, ( 1 ) in its columns, X ∈ ℝ× needs to be estimated, and N ∈ ℝ× where H ∈ ℝ× contains the observed spectral bands is the noise free signal which The denoising task is to estimate the unknown noise-free signal X.

LGOBE (P. Ghamisi) (P. Ghamisi) https://www.researchgate.net/profile/Behnood-Rasti (B. Rasti); http://www.pedram-ghamisi.com/ (P. Ghamisi) 0000-0002-1091-9841 (B. Rasti); 0000-0003-1203-741X © 2021 Copyright for this paper by its authors. Use permitted under Creative

X̂ = arg min 1

X 2 ‖H − X‖2 + (

X), ( 2 ) dard deviation of band . where determines the trade-of between the fidelity term and the penalty term ( X). We assume that the noise is uncorrelated spectrally, i.e., Ω = diag ( 12, 22, … , 2) is the noise covariance matrix where is the noise stan

2.1. Conventional Techniques

The conventional denoising approaches can be divided into two independent groups: full-rank and low-rank approaches.

2.1.1. Full-rank Approaches

Full-rank denoising approaches assume that X is fullrank. Since HSI is a collection of spectral bands, denoising approaches that have been developed for gray scale ing) can be applied on HSIs band by band. However, rich spectral information existing in HSIs is ignored in this way. On the other hand, HSI is also a combination of spectral pixels and, therefore, 1D signal denoising approaches such as multiple linear regression (MLR) proposed in [1] can be applied on spectral pixel vectors and, therefore, spatial information is ignored in this way. As a result, HSI denoising considerably benefits by exploiting both spatial and spectral information. 3D modeling and filtering such as 3D wavelets [2] and 3D (blockwise) nonlocal sparse denoising (Nonlocal SR) methods [3] are the earliest attempts for spatial-spectral denoising approaches.

Later on, penalized least squares exploiting spatial [4], denotes the noise. images (e.g., wavelet denoising and mean/median filterspectral [5], and spatial-spectral penalties [6] were pro- Since 2017, deep learning have been utilized for HSI posed for HSI denoising. An eficient edge-preserving denoising. The fundamental challenge for applying deep denoising approach is obtained with a prior ( X) in ( 2 ) learning for HSI denoising is the high-dimensionality of corresponding to the total variation (TV) of the signal HSIs. The high-dimensionality leads to a huge number [7]. of trainable parameters but, at the same time, the number

Several TV denoising approaches including Cubic TV of training samples is often limited in the remote sensing (CTV) [8, 9] and spatio-spectral TV [10] were adapted for community. The imbalance between the number of trainHSI denoising to capture the spectral correlation using ing samples and trainable parameters makes the network TV. training cumbersome to achieve a universal DL-based denoising technique for HSIs [23]. With regard to the 2.1.2. Low-Rank Approaches availability of training samples, HSI deep-learning-based denoising techniques can be split into two categories: Low-rank models assume that X is low-rank so to capture the high spectral correlation of HSI. The low-rank property can be applied in two diferent ways: ( 1 ) Using a low-rank model

H = FV + N,

(3) where F and V are of rank ( « min{, } ).

( 2 ) Using a low-rank penalty or constraint such as the nuclear norm in the cost function:

X̂ = arg min 1 ‖H − X‖2 + 1( X) + 2 ‖X‖∗ , (4)

X 2

2.2.1. Unsupervised/self-supervised

The unsupervised techniques do not utilize any training set and only rely on the observed image. One example of such approaches is deep HSI prior (HSI-DIP) introduced in [24]. HSI-DIP is based on deep image prior (DIP) [25], which utilizes a convolutional encoder-decoder network to implicitly induce a universal regularizer (the so-called image prior) in inverse problems including denoising. In [24], 2D convolution was extended to a 3D one to model the specific characteristics of HSIs, however, it was not as eficient as the 2D version. Self-supervised techniques are dependent on the observed image for training while they create their own training sets from the observed noisy image. In [26], a self-supervised (Zero-shot) denoising technique was introduced for HSIs. where ‖X‖∗ is the nuclear-norm of the matrix X, obtained by the sum of the singular values.

Several hyperspectral denoising approaches were proposed using Tucker3 decomposition [11, 12]. In [13], a wavelet-based low-rank model and ℓ1 regularization was proposed for HSI denoising (SVDSRR). An automatic hyperspectral restoration technique (HyRes) was proposed 2.2.2. Supervised in [14] using the ℓ1 penalized least squares and a low-rank These techniques demand training sets to train the netwavelet-based model. work. Most of the deep-learning-based denoising tech

In [15], the wavelet-based reduced-rank (WSRRR) model niques are supervised and their performances are highly was proposed for simultaneous HSI denoising and feature dependent on the availability of a high number of trainextraction using a non-convex optimization problem. ing data. In HSI denoising, the deep networks are trained

Low-rank TV regularization was also proposed in [16, in three ways: ( 1 ) using a database by collecting many 17] for both HSI denoising and feature extraction. patches [27], ( 2 ) using a database containing many real

Noise-adjusted image recovery using low-rank matrix HSIs such as ICVL [28] developed in [29], or (3) using approximation (NAIRLMA) was suggested in [18] exploit- a database by simulating HSIs based on RGB images using both low-rank and sparsity norms. Fast hyperspectral ing deep generative networks [30]. A spatial–spectral denoising (FastHyDe) proposed in [19] is also a low-rank deep residual CNN (HSID-CNN) was introduced in [27] technique that first project the HSI into a subspace and, in which 2D and 3D convolutional filters were utilized then, applies BM3D denoising [20] on the eigen images. to model spatial and spatial-spectral correlation in HSIs. Spectral linear unmixing techniques are also considered A 3D U-Net was introduced in [30] based on 2D convoas a low-rank HSI denoiser [21, 22]. lutional filtering in spatial direction and 1D convolutional ifltering in the spectral direction. A single model denois2.2. Deep learning-based Techniques ing CNN-based framework (SDeCNN) was introduced in [31]. In SDeCNN, spectral channels were extended along the spectral direction in the first place so that all spectral channels have centered by the same number of adjacent bands. Then, one band at a time was restored by applying the CNN on a downsampled HSI.

Deep learning-based denoising approaches are the stateof-the-art in the signal and image processing community but they have been mostly developed for RGB images.

Although they can usually be used for HSI denoising, they cannot model the specific characteristics of HSIs, e.g., low-rank and spectral dependency.

2.3. Experimental Results

Here, we compare the results of diferent hyperspectral The superiority of the low-rank techniques compared denoising algorithms including 3D Wavelet [2], FORPDN to the full-rank ones can be attributed to the spectral [5], SSTV [10], NAIRLMA [18], HyRes [14], HSI-DIP [24], redundancy of HSIs. Low-rank techniques capture the FastHyDe [19], and SDeCNN [31]. 3D wavelets, FORPDN, most variations of the signal by projecting it into a suband HyRes are parameter-free techniques. We set the space which helps to decorrelate the signal spectrally subspace dimension to 10 for FastHyDe. For HSI-DIP, from the noise. Denoising techniques such as HyRes all the hyperparameters and the layers of the encoder- and FastHyDe further denoise the signal spatially in its decoder architecture are adjusted as suggested in [25] to subspace. HSI-DIP is an unsupervised denoising techoptimize the performance. In the case of SDeCNN, we set nique, i.e., the network is trained using the observed data. the noise parameter to 1 since it gives the best results SDeCNN exploits only one CNN model which is trained for the simulated data while, for the real dataset, we used based on one dataset. Therefore, SDeCNN cannot perthe default value suggested by its authors i.e., = 20 . form well for all types of HSI data with diferent noise

The hyperspectral denoising techniques are applied on levels. We should note that, in the case of HSIs in para simulated noisy dataset and the results are compared ticular HSI denoising, the training is often challenging based on diferent levels of the noise power, i.e., PSNR=20, due to the absence or the limited number of training sets. 30, and 40 dB. The uncorrelated (the same variance for all Therefore, Low-rank-based HSI denoising techniques still bands) zero-mean Gaussian noise was added to a portion outperform the deep learning-based ones. of the Washington DC Mall dataset. Fig. 1 (a) and (b) compare the results of the techniques in terms of PSNR and spectral angle distance (SAD). The results are mean 3. Acknowledgments values over five experiments. The standard deviations are shown using error bars. The results confirm that the The work of Behnood Rasti was funded by the Alexanderlow-rank techniques used in the experiments outperform von-Humboldt-Stiftung/foundation. the other techniques in terms of both PSNR and SAD.

Additionally, HSI-DIP outperforms the full-rank conven- References tional techniques, i.e., 3D wavelet, FORPDN, and SSTV.

On the other hand, SDeCNN performs poorly.

For the real HSI denoising experiment, we apply all the denoising techniques on the Indian Pines dataset. The results of denoising for band 1 are compared visually in Fig. 2. The outcome of the visual comparison can be summarized as follows: NAIRLMA fails to restore band 1. 3D Wavelet restores band 1 with moderate visual quality. SDeCNN shows the poorest results. Band 1 is highly over-smoothed. We should note that SDeCNN was also applied to Indian Pines in [31], however, the authors removed the noisy bands from the dataset (resulting in a dataset having 206 bands) before applying the denoising techniques. Here, all 220 bands were used to evaluate the methods since the aim of HSI denoising techniques is to recover all the corrupted bands. SSTV, HyRes, and FastHyDe successfully and similarly reconstruct band 1. HSI-DIP also restores band 1 successfully, however, the restored band seems over-smoothed and blurred. FORPDN outperforms the other techniques visually. We should note that considering numerous bands existing in HSIs, this comparison might not reflect the performance of the techniques on the whole image. Band 1 is selected for comparison since it is noisy and it is often more challenging to recover the first and last few bands in HSIs due to the absence of the adjacent bands. [1] J. Bioucas-Dias, J. Nascimento, Hyperspectral subspace identification, IEEE Trans. Geos. Remote Sens 46 (2008) 2435 –2445. [2] B. Rasti, J. R. Sveinsson, M. O. Ulfarsson, J. A.

Benediktsson, Hyperspectral image denoising using 3D wavelets, in: IEEE IGARSS, 2012, pp.

1349–1352. [3] Y. Qian, M. Ye, Hyperspectral imagery restoration using nonlocal spectral-spatial structured sparse representation with noise estimation, IEEE Jour.

Sel. Top. App. Earth Obs. Remote Sens. 6 (2013) 499–515. [4] A. Zelinski, V. Goyal, Denoising hyperspectral imagery and recovering junk bands using wavelets and sparse approximation, in: IEEE IGARSS, 2006, pp. 387 –390. [5] B. Rasti, J. R. Sveinsson, M. O. Ulfarsson, J. A.

Benediktsson, Hyperspectral image denoising using first order spectral roughness penalty in wavelet domain, IEEE Jour. Sel. Top. App. Earth Obs. Remote Sens. 7 (2014) 2458–2467. [6] B. Rasti, J. R. Sveinsson, M. O. Ulfarsson, J. A.

Benediktsson, Wavelet based hyperspectral image restoration using spatial and spectral penalties, in:

Proc. SPIE, volume 8892, 2013, pp. 88920I–88920I–8.

[7] L. I. Rudin, S. Osher, E. Fatemi, Nonlinear total

30

PSNR (dB) (a) PSNR 3D Wavelet FORPDN SSTV NAIRLMA HyRes HSI-DIP FastHyDe SDeCNN 40 3D Wavelet FORPDN SSTV NAIRLMA HyRes HSI-DIP FastHyDe

SDeCNN 30 PSNR (dB) (b) SAD 40

variation based noise removal algorithms, Phys. D 60 (1992) 259–268. [8] Q. Yuan, L. Zhang, H. Shen, Hyperspectral image denoising employing a spectral-spatial adaptive total variation model., IEEE Trans. Geos. Remote Sens 50 ( 2012 ) 3660–3677. [9] H. Zhang, Hyperspectral image denoising with cubic total variation model, in: ISPRS Annals of

References

Photo. , Remote Sens . Spatial Inf. Sci. , volume I- 7, Image restoration for remote sensing: Overview

2012 , pp. 95 - 98 . and toolbox ( 2021 ). a r X i v : 2 1 0 7 . 0 0 5 5 7 . [10] H. K. Aggarwal , A. Majumdar , Hyperspectral im- [24] O. Sidorov , J. Y. Hardeberg , Deep hyperspectral

IEEE Geos. Remote Sens . Lett . 13 ( 2016 ) 442 - 446 . resolution, in: IEEE/CVF International Conference

doi:1 0 . 1 1 0 9 / L G R S . 2 0 1 6 . 2 5 1 8 2 1 8 . on Computer Vision Workshop (ICCVW), 2019 , pp. [11] A. Karami , M. Yazdi , A. Zolghadre Asli , Noise re- 3844-3851. doi:1 0 . 1 1 0 9 / I C C V W . 2 0 1 9 . 0 0 4 7 7 .

duction of hyperspectral images using kernel non- [25] D. Ulyanov , A. Vedaldi , V. Lempitsky , Deep

Sig. Proc. 5 ( 2011 ) 487 - 493 . puter Vision 128 ( 2020 ) 1867 - 1888 . URL: http://dx. [12] X. Liu , S. Bourennane , C. Fossati , Denoising of doi. org/10.1007/s11263-020-01303-4. doi:1 0 . 1 0 0 7 /

hyperspectral images using the parafac model and s 1 1 2 6 3 - 0 2 0 - 0 1 3 0 3 - 4 .

statistical performance analysis , IEEE Trans. Geos . [26] R. Imamura , T. Itasaka , M. Okuda , Zero-shot hy-

Remote Sens 50 ( 2012 ) 3717 - 3724 . doi:1 0 . 1 1 0 9 / T G R S . perspectral image denoising with separable image

2 0 1 2 . 2 1 8 7 0 6 3 . prior, in: 2019 IEEE/CVF International Conference [13] B. Rasti , J. R. Sveinsson , M. O. Ulfarsson , J. A. on Computer Vision Workshop (ICCVW), 2019 , pp.

Benediktsson , A new linear model and sparse reg- 1416-1420. doi:1 0 . 1 1 0 9 / I C C V W . 2 0 1 9 . 0 0 1 7 8 .

ularization, in: IEEE IGARSS, 2013 , pp. 457 - 460 . [27] Q. Yuan , Q. Zhang , J. Li , H. Shen , L. Zhang , [14] B. Rasti , M. O. Ulfarsson , P. Ghamisi , Automatic Hyperspectral image denoising employing a spa-

low-rank modeling , IEEE Geos. Remote Sens. Lett. work , IEEE Trans. Geos. Remote Sens . 57 ( 2019 )

PP ( 2017 ) 1 - 5 . doi:1 0 . 1 1 0 9 / L G R S . 2 0 1 7 . 2 7 6 4 0 5 9 . 1205 - 1218 . doi:1 0 . 1 1 0 9 / T G R S . 2 0 1 8 . 2 8 6 5 1 9 7 . [15] B. Rasti , J. Sveinsson , M. Ulfarsson , Wavelet-based [28] B. Arad , O. Ben-Shahar , Sparse recovery of hyper-

52 ( 2014 ) 6688 - 6698 . 2016 , pp. 19 - 34 . [16] B. Rasti , J. R. Sveinsson , M. O. Ulfarsson , Total [29] Y. Chang , L. Yan , H. Fang , S. Zhong , W. Liao, Hsi-

in: IEEE IGARSS , 2014 , pp. 4644 - 4647 . lutional neural network , IEEE Trans. Geos . Re[17] B. Rasti , Sparse Hyperspectral Image Modeling and mote Sens . 57 ( 2019 ) 667 - 682 . doi:1 0 . 1 1 0 9 / T G R S .

Restoration , Ph.D. thesis, Dep. of Electrical and 2 0 1 8 . 2 8 5 9 2 0 3 .

Computer Engineering , University of Iceland, 2014 . [30] W. Dong , H. Wang , F. Wu , G. Shi , X. Li , Deep spa[18] W. He , H. Zhang , L. Zhang, H. Shen, Hyperspectral tial-spectral representation learning for hyperspec-

rank matrix approximation , IEEE Jour. Sel. Top . 5 ( 2019 ) 635 - 648 . doi:1 0 . 1 1 0 9 / T C I . 2 0 1 9 . 2 9 1 1 8 8 1 .

App. Earth Obs . Remote Sens . 8 ( 2015 ) 3050 - 3061 . [31] A. Mafei , J. M. Haut , M. E. Paoletti , J. Plaza , L. Bruz-

doi:1 0 . 1 1 0 9 / J S T A R S . 2 0 1 5 . 2 3 9 8 4 3 3 . zone, A. Plaza , A single model cnn for hyperspectral [19] L. Zhuang , J. M. Bioucas-Dias , Fast hyperspectral image denoising , IEEE Trans. Geos. Remote Sens. 58

image denoising and inpainting based on low-rank ( 2020 ) 2516 - 2529 . doi:1 0 . 1 1 0 9 / T G R S . 2 0 1 9 . 2 9 5 2 0 6 2 .

App. Earth Obs . Remote Sens . 11 ( 2018 ) 730 - 742 .

doi:1 0 . 1 1 0 9 / J S T A R S . 2 0 1 8 . 2 7 9 6 5 7 0 . A. Online Resources [20] K. Dabov , A. Foi , V. Katkovnik , K. Egiazarian , Im-

age Denoising by Sparse 3- D Transform-Domain The sources for the denoising toolbox to reproduce the

Collaborative Filtering , IEEE Trans. Imag. Proc. 16 results are available via

(Agu.) 2080 - 2095 . [21] B. Rasti , B. Koirala , P. Scheunders , P. Ghamisi , How

boost each other , Remote Sens . 12 ( 2020 ) 1728 . URL:

http://dx.doi.org/10.3390/rs12111728. doi:1 0 . 3 3 9 0 /

r s 1 2 1 1 1 7 2 8 . [22] D. Cerra , R. Müller , P. Reinartz , Noise reduction in

IEEE Geos. Remote Sens . Lett . 11 ( 2014 ) 109 - 113 . [23] B. Rasti , Y. Chang , E. Dalsasso , L. Denis , P. Ghamisi, • https://github.com/BehnoodRasti/Hyperspectral-Image-

Denoising-Toolbox-V2.