A New Optimization Model for the Restoration of the Deteriorated Hyperspectral Images

Abstract

Hyperspectral imaging technology has great role in performing computer vision tasks eficiently. However the acquired hyperspectral images (HSIs) are contaminated by diferent types of noises and other unwanted signals. This paper proposes a new tensor svd based low rank decomposition together with spatial spectral total variation (SSTV) regularization for removing the noise artefacts in HSIs. The proposed optimization model uses tensor decomposition to express the correlation among the diferent frequency bands. The sparse noise is detected using a 1 norm, and in addition, a Frobenius norm is added to remove heavy Gaussian noise from the images. A SSTV norm is added to preserve the piecewise smoothness structure in the spatial and spectral domains. An eficient solution for the optimization problem is developed based on the alternating direction method of multipliers (ADMM). From the experiments conducted on noisy HSIs, it can be observed that our method achieves better results compared to the already existing ones.

Keywords

• eol>Hyperspectral image (HSI)
• low-rank tensor decomposition
• denoising
• spatial spectral total variation (SSTV)

1. Introduction

hyperspectral images can be done in 4 modes- line scanning, area scanning, point scanning, and single shot [ 2 ]. In a pushbroom based hyperspectral system, the sensor captures spectral information at each line, and by moving the camera or the object, the system captures the entire spatial region[ 2 ]. The major components of a pushbroom hyperspectral imaging system are: illumination, camera, objective lens, transportation plate, and a data processing unit, as shown in Fig.1 [ 3 ]. The spectral range produced by the system depends on illumination conditions, camera, and the typical range of a VNIR system is 400 to 1000 nm. Even if HSI is a significant PC

Illuminant

Fruit Moving platform

VNIR

Hyperspectral camera

Illuminant

Spectralon tile

Translator stage Motor non-invasive quality assessment technique, real HSIs always sufer from various degradations. The anomalous observations can be due to several reasons, such as the dysfunction and noise in the sensor or from diferent stages in the workflow, shape and geometry of the scene, and the radiation technique [ 4 ]. In this paper, we try to develop a new model for removing specific noise artefacts afecting the HS images.

The pushbroom sensor used for image acquisition produces stripes, and random noise [ 5 ]. The random noise can be viewed as an adaptive white Gaussian(AWGN) random process with variance 2 and mean zero. Dead detectors will give rise to single line drop-out for pushbroom sensors [ 6 ]. The neighbouring elements in the CCD(Charge Coupled Device) array may have sensitivity variations, which causes vertical stripe noise [ 7 ]. HSIs also sufer from impulse noise [ 8 ]. These noises can afect the subsequent internal and external quality evaluation and defect detection processes. Improving the HSI quality merely through a hardware scheme is unsustainable and impractical. Therefore, suitable image pre-processing techniques are to be applied to obtain a high-quality HSI before subsequent applications.

HS imaging was actually developed for remote sensing applications, and so far, a huge number of proposals have been made in the HSI denoising. These techniques can be broadly classified into five categories [ 9 ]. In the first category, each band in the HSI is treated as an image, and then commonly used two dimensional (2D) denoising methods are applied in each band to remove noise [ 10 ],[ 11 ]. But the results are not promising since the band-wise processing does not consider the strong correlations between bands. Another method is to treat the HSI as a multidimensional data cube, and the volumetric data denoising methods can accomplish the denoising [ 12 ],[ 13 ]. However, the results are unsatisfactory because these methods do not consider the correlation among the bands. Therefore, it is necessary to consider the spatial and spectral information simultaneously to enhance the denoising quality. The third type of HSI denoising method combines spatial and spectral information, and several algorithms have been proposed based on this concept.e.g. [ 14 ], [ 15 ]. The results obtained from this fusion method are promising compared with single view(either spatial or spectral) methods. A few transform domain-based techniques were also proposed in this category [ 16 ], [ 17 ].

The fourth type of HSI denoising method falls under the category of low rank based techniques. In [ 18 ], a randomized singular value decomposition is introduced and a noise adjusted iterative framework is proposed for low rank matrix approximation (NAILRMA). A SSTV based low Tucker rank model (LRTDTV) was proposed to preserve the spectral signatures of HSI in [ 19 ]. The method proposed by Chen et al.[ 20 ] is based on weighted group sparsity regularization, and this concept is incorporated for low rank tucker decomposition (LRTDGS), in which results were shown to be improved in comparison with the previous TV based methods. Zheng et al. [ 21 ] proposed the fibered rank and suggested the convex (3DTNN) and non-convex models (3DLogTNN) to separate clean HSI from noisy images.

Recently, many deep learning based algorithms have been proposed, and their performance is encouraging[ 22 ], [ 23 ]. But their results are highly dependent on the quality and volume of training data which limits their application.

Restoration of HS images for remote sensing applications has been well studied in the literature. But the researchers have less focused on restoring the images for close range applications. Moreover, the algorithms used for remote sensing applications may not be suitable for HSI restoration of data captured at proximity range. This is evident from Fig.2 wherein one of the recently proposed remote sensing HSI denoising techniques fails to restore the original fruit image. Hence we propose and test an algorithm specifically for restoring deteriorated hyperspectral images for close range applications.

(a) (b)

The low rank based HSI restoration techniques can be broadly classified as matrix based and tensor based approaches. Even though several methods have been proposed based on low rank matrix modelling for removing mixed noise, these methods failed to utilise the correlation in spatial and spectral modes efectively, leading to suboptimal denoising results under severe noisy conditions. So many recent research works included the direct tensor modelling techniques, and it has been found that these techniques are superior to matrix based techniques in computing higher order data. Such studies motivated us to propose a new optimisation model for restoring noise-contaminated hyperspectral images using direct tensor modelling techniques exploiting the spatial and spectral information.

Our Contributions: . As per our knowledge, this is the first efort to realise a tensor based low rank(LR) and SSTV model for the removal of noise artefacts from the HSI data captured at proximity range. . Further, 1 norm and Frobenius norms are used to address the sparse noise and heavy

Gaussian noise in the images. . The formulated optimisation problem is decomposed into several subproblems and solved using the Alternating Direction Method of Multipliers (ADMM).

. The algorithm is implemented successfully on the fruit HSI data with simulated noise. The rest of the paper is organised as follows. Section 2 explains the tensor based low rank and SSTV model and the optimisation solution in detail. The experimental results are presented in section 3. Finally, the conclusions are derived in section 4.

2. Proposed Model

A noisy HSI cube ℋ may be represented as,

ℋ = + ℰ where represents the clean HS image and ℰ represents noise artefacts. The aim of HSI restoration task is to obtain the noise-free HS image from the noise-contaminated image ℋ. The noise term includes Gaussian noise and the sparse noise . Dead pixels, stripes and impulse noise fall into the category of sparse noise. Based on this, the degraded HSI can be modelled as,

Using the TRPCA (Tensor Robust Principal Component Analysis) model [ 24 ], the image can be expressed as a combination of a rank function which describes the low rank property and a ℓ0 norm to represent the sparse noise.

ℋ = + + argmin rank( ) + ‖‖0 where ℋ, ,, and are 3order tensors. The low rank approximation can be obtained using tensor singular value decomposition algorithm[ 25 ]. The objective of using the TV regularization is to take advantage of the piecewise smoothness property in both the spectral and spatial domains. The widely used 2D TV regularizer consider only the spatial smoothness structure of HSI. So we also use a SSTV regularizer to explore the spectral smoothness.

‖ ‖SSTV := ∑︁ w1|pi,j,k − i,j,k pi,j,k-1| + w2|pi,j,k −

pi,j-1,k| + w3|pi,j,k − pi-1,j,k| is also added to the rank-constrained RPCA, the model will become, where ,,, is the (i, j, k)ℎ entry of . The term (l = 1, 2, 3) is the weight along the lℎ mode of . These weights decide the strength of regularization. When the SSTV regularization By adding some auxiliary variables Eq.(6) can be rewritten as, argmin ‖ ‖⊛ + ‖‖1 + ‖ ‖SSTV, .. argmin ‖ ‖⊛ + ‖‖1 + ‖‖F2 + ‖ℱ ‖1,

ℋ = + + , = , () = ℱ where (·) is the third order weighted diference operator. It can be given by (·) = [ × (·); × (·); × (·)] where , , and represents the first order diference operators in the diferent orientations of a 3D HSI. The optimization problem in Eq.(7) can be rewritten using the augmented Lagrangian multiplier (ALM) method: argmin ‖ ‖⊛ + ‖‖1 + ‖‖F2 + ‖ℱ ‖1 + ⟨1, ℋ − − − ⟩ + ⟨2, − ⟩ + 2 ‖ − ‖ F

2 + ⟨3, () − ℱ ⟩ +

2 ‖() − ℱ ‖ F 2 + 2 ‖ℋ − − − ‖ 2 F update +1 =argmin ‖ ‖⊛ + ⟨k1, ℋ − − k − k ⟩ + + 2 ‖ℋ − − 2 ‖ − ‖F k

2 =argmin ‖ ‖⊛ + 2( 2 )‖ − ℳ‖ F − k‖F2 + ⟨2, − k ⟩ (4) (5) (6) (7) (8) (9) =argmin ‖ ‖F2 + ⟨1, ℋ − − k+1 − ⟩ where

update

update update

Hz = k+1 + (ℱ ) + 2 −

(3) Tz = 2|fftn()|2 + 2|fftn()|2 + 2|fftn()|2 k+1 = ifftn ︂( fftn() )︂

I + 2

︂( 2 +

‖ℋ − =argmin ‖‖1 + = ℋ − k+1 − − 2 ‖ − − k+1 ‖F (ℋ − k1 )︂ k

+ +

‖ℋ − =argmin ( + = (ℋ − k+1 ⃦ 2 ⃦

⃦ − k+1 + 2

− )⃦ − k+1 − ‖ F (ℋ − k+1) + k1 k+1 − − k + 1

2 )‖F k+1

− + 2 k+1) + k1 ⃦⃦ 2 ⃦ ⃦ F k+1 2 + 2 2 +1 =argmin ⟨2, k k+1 − ⟩ ‖ k+1 − ‖ F

2

k + ⟨3, ( ) − ℱ k ⟩ + ‖( ) − The solution to the above equation can be obtained from the following linear equation. ( I + ) = +1 + (ℱ ) + 2 − (3) where indicates the adjoint operator of . We adopt the Fourier Transform to solve Eq.(12). (10) (11) (12) (13) ℱ update The updates of the multipliers are given by ℱ +1 =argmin ‖ℱ ‖1 + ⟨k3, (

k+1) − ℱ ⟩ ℱ 2 ℱ + ‖( k+1) − ℱ ‖ F

2 2 =argmin ‖ℱ ‖1 +

‖ℱ − (( = (( +1) + 3 ) k+1) + k3

)‖F2 k1+1 = k1 + (ℋ − k2+1 = k2 + ( k+1

k+1 − − k+1) k3+1 = k3 + (( k+1) − ℱ k+1 k+1) − k+1) (14) (15) Algorithm 1: HS image denoising

weights [ , , ] 2 Output: Denoised HSI 4 = 106, = 1.5 and = 0 3 Initialize: = = = = 0,1, 2, 3 = 0, 5 while not converged do 6 7 8 9 10 11 end update 1, 2, 3 via Eq. (15) update parameter :=min(, check for the convergence criteria

) update , , , , ℱ via Eqns. (9),(10),(11),(12),(13) and (14)

‖ℋ‖2 ‖k− k+1‖2 ≤ 1 Input: Observed noisy HSI ℋ, convergence criteria , the parameters of regularisation , , and the 3. EXPERIMENTAL RESULTS AND DISCUSSION The proposed method is compared with five other best-performing denoising techniques to show the efectiveness in HSI restoration. These methods include LLRGTV [ 26 ], 3DTNN[ 21 ], 3DLogTNN [ 21 ], NAILRMA [ 18 ], and LRTDTV [ 19 ]. The experiments for all the compared methods are conducted using the same parameters given by the authors in the respective papers to ensure optimum performance. An Intel Xeon CPU@ 3.50 GHz with 64-GB RAM is used to conduct the experiments in MATLAB R2022a 3.1. Dataset and Experiments: Hyperspectral images with diferent spectral ranges can be used to detect the quality attributes and defects of horticultural products[ 27 ]. The dataset for this study is obtained using a camera HySpex-VNIR-1800 which has a spectral sensitivity from 400 nm to 1000 nm and spectral sampling of 3.18 nm [ 3 ]. This VNIR push broom scanner records 186 spectral bands and 1800 pixels across the field of view. It uses a polariser to avoid specular reflection. In the present case study, we used the fruits cherry and strawberry. Since the size of the dataset is computationally intensive, it is cropped to 500× 500 pixels for cherry and 600× 700 pixels for strawberry. Real HSIs are afected by a variety of noises, such as Gaussian noise and sparse noise in diferent amounts. In order to simulate real noise conditions, we consider diferent combinations of these noises in varying proportions. Thus we can define the following noise cases as follows.

Noise case 1: Gaussian noise + Impulse noise: Gaussian noise and salt and pepper noise with equal distribution are added to each and every band. The variance of the Gaussian noise is set as 0.2 with zero mean, and the percentage of the impulse noise added is 0.2.

Noise case 2: Gaussian noise + Impulse noise + Dead lines: Here, the variance of Gaussian noise and percentage of impulse noise were fixed as 0.15. Besides this, we add dead lines whose width varies randomly between 1 and 3. The dead lines are added to bands ranging from 101 to 140, and the number of lines randomly varies between 3 and 10.

Noise case 3: Gaussian noise + Impulse noise + Dead lines + Stripes: Here, in addition to the noises as in case 2, we add some stripes also to bands ranging from 91 to 130. The number of stripes randomly varies from 30 to 40. The variance of the Gaussian noise was 0.05 with zero mean, and the percentage of the impulse noise was 0.05.

The visual quality and quantitative metrics for the above noise cases are tested for all methods used for comparison.

Visual quality comparison: The denoising results for case 1 and case 3, of band 120 are shown in Figs. 3,4,5 and 6. From Figs. 3(b),4(b),5(b) and 6(b) it can be observed that the original HSIs are severely afected by a combination of various noises. In all the noise cases, it is found that the proposed method gives the best performance. This demonstrates that the low rank minimisation, together with the TV regularisation, perform well for HSI restoration. Quantitative comparison: For quantitative evaluation, we use the metrics such as the mean peak signal-to-noise ratio (MPSNR) and the mean structural similarity index (MSSIM). The higher the values for the PSNR and SSIM, the better the restoration results.

MPSNR = 1 ∑︁ =0 MSSIM = 1 ∑︁ =0 where and are the PSNR and SSIM values for the th band, respectively. Table 1 and Table 2 show the restoration results of cherry and strawberry respectively for all methods considered for comparison. The psnr and ssim values obtained for the diferent noise cases clearly shows that the proposed method gives better restoration results than the other methods. 3.2. Discussion: Parameter setting: There are several parameters in the algorithm that need to be carefully adjusted in order to get optimum restoration results. The sparsity regularization parameter can be selected as = 100x √ , where a is the height and b is the width of a single HSI band, and C is a parameter for tuning. In all the simulated data experiments, the value of C is fixed as 10. is the Frobenius norm regularization parameter, which is selected as the inverse of the variance of Gaussian noise. The SSTV regularization parameter can be fixed as a constant equal to 1. The weights of the SSTV are chosen as 1 along the spatial mode and a value in the range of 0 to 1 in the spectral mode. The typical value used in the simulation experiment is [1,1,0.8].

Computational speed: The running time of diferent models are compared on the simulated data . Table 3 and Table 4 show the details for diferent methods. It can be observed that the NAILRMA method has the lowest computation time among all the compared methods. But, the quantitative and visual comparison results of this method are not good. The converging time for LLRGTV, 3DTNN and 3DLogTNN methods are much higher than the other methods since they require running an expensive iterative algorithm. The computation time of the proposed method is relatively higher than some other methods such as LRTDTV and NAILRMA but significantly lower than LLRGTV, 3DTNN and 3DLogTNN.

4. Conclusion

A low rank tensor based model is proposed for removing noise artefacts in the line scan-based hyperspectral images. Additionally an SSTV regularization is used, which preserves the spatial smoothness and spectral correlation. We have used 1 norm to detect the sparse noise efectively, and in order to tackle the heavy Gaussian noise conditions which may occur in real situations, we have also considered a Frobenius norm. A new algorithm based on the ALM method is designed to solve the resulting nonconvex optimization model. The algorithm is implemented and tested successfully on the fruit HSI data with simulated noise. The experiments show that the proposed method performs better than some popular methods used for comparison in terms of quantitative evaluation and visual comparison.

References

[1] M. J. Khan , H. S. Khan , A. Yousaf , K. Khurshid , A. Abbas , Modern trends in hyperspectral image analysis: A review , IEEE Access 6 ( 2018 ) 14118 - 14129 .

[2] Y. Lu , Y. Huang , R. Lu , Innovative hyperspectral imaging-based techniques for quality evaluation of fruits and vegetables: A review , in: Appl. Sci., volume 7 , 2017 .

[3] B. Devassy , S. George , Estimation of strawberry firmness using hyperspectral imaging: a comparison of regression models , Journal of Spectral Imaging 10 ( 2021 ) a3 .

[4] M. Vidal , J. M. Amigo , Pre-processing of hyperspectral images. essential steps before image analysis , Chemometrics and Intelligent Laboratory Systems 117 ( 2012 ) 138 - 148 .

[5] W. Kang , S. Yu , D. Seo , J. Jeong , J. Paik , Push-broom-type very high-resolution satellite sensor data correction using combined wavelet-fourier and multiscale non-local means ifltering , Sensors 15 ( 2015 ) 22826 - 22853 .

[6] M. Bouali , S. Ladjal , Spectral Inpainting for the Restoration of Missing Data from Multispectral Satellite Sensors: Case study on Aqua MODIS Band 6 , 2011 .

[7] L. Gómez-Chova , L. Alonso , L. Guanter , G. Camps-Valls , J. Calpe , J. Moreno , Correction of systematic spatial noise in push-broom hyperspectral sensors: application to chris/proba images , Appl. Opt . 47 ( 2008 ) F46 - F60 .

[8] S. Wang , Z. Zhu , R. Zhao , B. Zhang. , Hyperspectral image denoising via nonconvex logarithmic penalty , Mathematical Problems in Engineering ( 2021 ).

[9] F. Yang , X. Chen , L. Chai , Hyperspectral image destriping and denoising using stripe and spectral low-rank matrix recovery and global spatial-spectral total variation , Remote Sensing 13 ( 2021 ).

[10] A. Buades , B. Coll , J. Morel , A non-local algorithm for image denoising , in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , volume 2 , 2005 , pp. 60 - 65 vol. 2 .

[11] K. Dabov , A. Foi , V. Katkovnik , K. Egiazarian , Image denoising by sparse 3-d transformdomain collaborative filtering , IEEE Tran. on Image Process . 16 ( 2007 ) 2080 - 2095 .

[12] Y. Qian , Y. Shen , M. Ye , Q. Wang , 3 -d nonlocal means filter with noise estimation for hyperspectral imagery denoising , in: 2012 IEEE International Geosci. and Rem. Sens. Symposium , 2012 , pp. 1345 - 1348 .

[13] M. Maggioni , V. Katkovnik , K. Egiazarian , A. Foi , Nonlocal transform-domain filter for volumetric data denoising and reconstruction , IEEE Trans. on Image Process . 22 ( 2013 ) 119 - 133 .

[14] Q. Yuan , L. Zhang , H. Shen, Hyperspectral image denoising employing a spectral-spatial adaptive total variation model , IEEE Trans. on Geoscience and Remote Sensing 50 ( 2012 ) 3660 - 3677 .

[15] X. Huang , B. Du , D. Tao , L. Zhang, Spatial-spectral weighted nuclear norm minimization for hyperspectral image denoising , Neurocomputing 399 ( 2020 ).

[16] G. Chen , S.-E. Qian, Denoising of hyperspectral imagery using principal component analysis and wavelet shrinkage , IEEE Transactions on Geoscience and Remote Sensing 49 ( 2011 ) 973 - 980 .

[17] B. Rasti , J. R. Sveinsson , M. O. Ulfarsson , J. A. Benediktsson , Hyperspectral image denoising using 3d wavelets , in: 2012 IEEE International Geoscience and Remote Sensing Symposium , 2012 , pp. 1349 - 1352 .

[18] W. He , H. Zhang , L. Zhang, H. Shen, Hyperspectral image denoising via noise-adjusted iterative low-rank matrix approximation , IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 8 ( 2015 ) 3050 - 3061 .

[19] Y. Wang , J. Peng , Q. Zhao , Y. Leung , X.-L. Zhao , , D. Meng ., Hyperspectral image restoration via total variation regularized low-rank tensor decomposition , IEEE Journal of Selected Topics in Applied Earth Observation and Remote Sensing . 11 ( 2018 ) 1227 - 1243 .

[20] Y. Chen , W. He , N. Yokoya , , T. Huang., Hyperspectral image restoration using weighted group sparsity-regularized low-rank tensor decomposition , IEEE Trans. on Cybernetics . 50 ( 2020 ) 3556 - 3570 .

[21] Y. Zheng , T. Huang , X.-L. Zhao , T. Jiang , T. Ma, , T. Ji, Mixed noise removal in hyperspectral image via low-fibered rank regularization , IEEE Trans. Geosci. Remote Sens . 58 ( 2020 ) 734 - 749 .

[22] Y. Gou , B. Li , Z. Liu , S. Yang , X. Peng , Clearer: Multi-scale neural architecture search for image restoration , in: H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan , H. Lin (Eds.), Advances in Neural Information Processing Systems , volume 33 , Curran Associates , Inc., 2020 , pp. 17129 - 17140 .

[23] Q. Yuan , Q. Zhang , J. Li , H. Shen , L. Zhang, Hyperspectral image denoising employing a spatial-spectral deep residual convolutional neural network , IEEE Trans. on Geosci. and Rem. Sens . 57 ( 2019 ) 1205 - 1218 .

[24] C. Lu , J. Feng , Y. Chen , W. Liu , Z. Lin , S. Yan , Tensor robust principal component analysis with a new tensor nuclear norm , IEEE Trans. on Pattern Analysis and Machine Intelligence 42 ( 2020 ) 925 - 938 .

[25] M. E. Kilmer , K. Braman , N. Hao , R. C. Hoover ., Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging , SIAM J. Matrix Anal. Appl . 34 ( 2013 ) 148 - 172 .

[26] W. He , H. Zhang , H. Shen , L. Zhang, Hyperspectral image denoising using local low-rank matrix recovery and global spatial-spectral total variation , IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11 ( 2018 ) 713 - 729 .

[27] A. Gowen , C. O'Donnell , P. Cullen , G. Downey, J. Frias. , Hyperspectral imaging - an emerging process analytical tool for food quality and safety control , Trends in Food Science & Technology 18 ( 2007 ) 590 - 598 .