A Wavelet and HSV Pansharpening Technology of High Resolution Satellite Images

Abstract

High resolution satellite images are used to monitor environmental changes, map-making and military intelligence and forecast natural disasters. Nowadays, these contain spatial dissimilarities due to differences in their radiometry resolution, spectral characteristics and time delay from high resolution satellite sensors (multispectral and panchromatic). The use of pansharpened high spatial resolution images significantly increases the possibility of thematic recognition. Pansharpening is a technique that is used to combine the spatial details of a panchromatic image with the the several spectral bands of a lower resolution multispectral image. To date there have proposed a large number of fusion methods. However, most available methods are not effective for the latest very high resolution, such as WorldView-3 satellite imagery. Most methods for increasing spatial resolution lead to artifacts - objects that are not present in the original scene, but that appear in the resulting image. In this paper, we present the pansharpening technology of high resolution satellite images, using integration of bicubic interpolation, color system HSV and wavelet-transform. The aim of the proposed technology is to obtain the high resolution multispectral satellite image after the previous geometric correction of primary multispectral images and optimal wavelet decomposition into approximation and detail coefficients according to the chosen information value function linear forms. The proposed technology was verified by a number of different satellite data. The experimental evaluations are carried out on WorldView-3 images. Visual and quantitative analyses show that our presented technology can achieve high spectral and spatial quality and outperforms some existing pansharpening methods.

Keywords

• Pansharpening
• Satellite Image
• High Resolution
• Panchromatic
• Multispectral
• Wavelet Transform
• Resampling

-

High resolution satellites, such as WorldView-2,3, provide very valuable data about the Earth, e.g., for urban damage detection, environmental monitoring, weather forecasting, map-making and military intelligence [ 1-3 ]. The satellite images are characterized by their spatial, spectral, radiometric and temporal resolutions. But for more practical applications, used only spatial and spectral resolutions are considered. Generally, satellites take various images from different frequencies in the visual and nonvisual ranges called as monochrome images. Most modern satellite systems that monitor the Earth have the ability to obtain multispectral (MUL) and panchromatic (PAN) images of different spatial resolutions. All things being equal, panchromatic images have a higher spatial resolution. Based on the frequency range each monochrome image contains various information about the object. Each monochrome image is represented as a band [ 4 ]. Multispectral image of high spatial resolution sensor contains four (Red, Green, Blue and near-infrared) or eight bands (Coastal, Blue, Green, Yellow, Red, Red Edge, Near-Infrared 1, NearInfrared 2). The combination of these bands produces a new color image [ 5 ]. 2

State of Art

At present, there are different pansharpening methods which can generally be divided into: Brovey-transform, principal component analysis, independent component analysis, Gram-Schmidt, intensity-hue-saturation transform (IHS) [ 6-13 ]. The typical algorithm for component substitution fusion technique is IHS transform fusion algorithm, which provides good quality visual high resolution multispectral image, but spectral distortion occurs [ 14 ]. Rahmani et al. proposed a modified IHS pansharpening method [ 14 ]. An image adaptive coefficient for IHS was found to obtain a more accurate spectral resolution. Work [ 15 ] proposed a pansharpening with multiscale normalized nonlocal means filter. This filter computes each pixel value as the weighted average of all pixels over a sliding window, and the pixel weight depends upon the distance from the center pixel. The decomposed high frequency details are added into each band of multispectral image and the final smooth fused image is obtained. Almost all these approach are fast and easy to implement but suffer from spectral distortion because the PAN image does not cover exactly the same spectrum band as that of the MUL image; and not efficient for the latest image generation.

In the last decade, image fusion techniques based on Multi Resolution Analysis methods, like wavelet transform [ 12, 16 ] and “à-trous” wavelet transform have become significant due to their ability to capture the information present at different scales. It is accomplished by using wavelet basis functions due to its desirable properties such as multiscale decomposition and space frequency localization [17]. In [18] component substitution fusion method is proposed to reduce color distortion. Work [19] proposed method divided multispectral and panchromatic images into several pixel groups by k-means algorithm. Then, the panchromatic image was estimated by a weighted summation of MUL bands and the fused image was generated by ratio enhancement [19]. In [20] authors suggested pansharpening algorithm using a guided filter that has good properties such as edge-preserving and structure transferring. The underlying idea of the approach in [21] is to consider the spectral difference of each pixel between multispectral image and panchromatic image, and to adaptively inject the Pan details into the MUL image. An improved image fusion method was proposed through the improvement of fused spectra of of mixed pixels [22]. In paper [23] authors consider the application of nonlinear image decomposition schemes based on morphological operators to data fusion. Work [24] proposes a new regularized modelbased pan-sharpening method for the images with local dissimilarities. An adjustment matrix is introduced into the global spatial similarity regulariser to reduce the effect of the contrast inversion [24]. Recent research has shown that the deep neural networks have obtained superior performance in pansharpening image [25-29]. However, the neural network methods take more time than traditional fusion methods.

The analysis of the existing pansharpening approaches showed that most methods for increasing spatial resolution lead to artifacts. Objects that are not present in the original scene, but that appear in the resulting image. Most of existing pansharpening methods generally introduce spectral distortions. 3

Pansharpening Technology

We propose an efficient pansharpening technology of high resolution satellite images using integration of bicubic interpolation, HSV and wavelet-transforms. The technology scheme is shown in the Fig. 1.

The main processing steps are: 1. Uploading the high resolution images received from satellite WorldView-3: panchromatic – PAN, multispectral: MUL in true-color composition (R, G, B) and NIR in color composition (NIR, B, R). 2. Resampling MUL and NIR based on bicubic interpolation [30]: (1) (2) where aij – coefficient; Pij – intensity of the image being scaled. 3. A characteristic feature of many images obtained in real scanner satellite system is a significant proportion of dark areas and the relatively small number of areas with high brightness. That is why one of the first steps of the algorithm is histogram equalization images. Discrete transformation brightness scale is as follows: where zi′ – the value of the conversion scale brightness corresponding to the brightness output scale;

p(zk ) – normalized histogram brightness of the original image ( k = 0...255 ). 4. Transform the multispectral image (MUL) from RGB components into hue– saturation-intensity (HSV) components:

3 3 v (x,y) = ∑ ∑ aij ⋅ Pij , i=0 j=0

i zi′ = zm ∑ p(zk ) ,

k =0 VV1  =  1136 V2   12 (3) (4) (5) 5. The next stage is applying wavelet transform (Fig.2) which is divided into the next stages: 5.1. Decompose the PAN into approximation coefficients (LL) and detail coefficients (LH, HL, and HH, the image information including vertical, horizontal and diagonal features) to the fourth decomposition level of the discrete wavelet transform (DWT) of the bior 2.2 class:

N PAN → PLNL + ∑ (PLiH + PHi L + PHi H ) ,

i where, PLNL – approximation coefficient at level N; PLiH , PHi L , PHi H – horizontal, vertical, diagonal coefficients at level i respectively. 5.2. Apply DWT to luminance Vm-component of the multispectral image. The images are subjected to the fourth level of decomposition, resulting in each of the images are approximate and detailing matrix coefficients:

N Vm → VmLNL + ∑ (VmLiH + VmHiL + VmHiH ) ,

i where, VmLNL – approximation coefficient at level N; VmLiH , VmHiL , VmHiH – horizontal, vertical and diagonal coefficients at level i. 5.3. Apply the concept of DWT to the NIR image. Matrix NIRLL holds the approximation coefficient. Matrices NIRLH, NIRHL, and NIRHH keep the detail coefficients:

N NIR → NIRLNL + ∑ (NIRLiH + NIRHiL + NIRHiH ) , i (6) where, NIRLNL – approximation coefficient at level N; NIRLiH , NIRHiL , NIRHiH – horizontal, vertical and diagonal coefficients at level i.

5.4. After decomposition, substitution has been done by placing Vm bands approximation in PAN approximation at each level and NIR bands approximation in PAN approximation at each level. For each Vm, NIR band and PAN, single fused image coefficients are obtained. Similarly for each level, two fused image coefficients have been obtained. The coefficients merging rule is:

N N VmLNL + ∑(PLiH + PHiL + PHiH ) → Vmw; NIRLNL + ∑(PLiH + PHiL + PHiH ) → NIRw.

i i 5.5. Applying the Biorthogonal wavelet transform (IDWT) to matrix obtained in the last step for obtaining the new intensity components Vmv and NIRw. IDWT performs the inverse DWT, which allows reconstructing new intensity components where the features from the PAN and those from the initial intensity component have been integrated. After inverse wavelet transform, two new fused images have been obtained in each level. 6. Reverse transform from the HSV in the RGB color space, choose Hm and Sm components of multichannel images and the resulting Vmw after wavelettransform. 7. Taking the new RGB channels from the true-color composite and the new NIRw channel from the false-color composite. The result is four-channel image high spatial resolution. 4 4.1

Results

Visual Analysis

The testing WorldView-3 data set consists of 8-band multispectral data with 1.24 m spatial resolution and panchromatic image with 0.31 m spatial resolution. Fig. 3 shows 400*400 detail of the whole scene, which constitutes buildings, grass, trees and roads. The MUL subscene image in RGB composition (Band 5-3-2) is shown in Fig 3a, in NIR color composition (Band 7-2-5) is shown in Fig 3b and full-resolution panchromatic image is shown in Fig.3c. The results obtained from Brovey-transform, Gram-Schmidt methods and the proposed technology are reported in Figs. 3(c)-3(f), respectively. A visual comparison of the results makes it possible to assert that the spatial resolution of the original multispectral data is improved. Buildings and roads in the resulting image are much sharper than in the original image. The result obtained from proposed technology is shown the spatial details appear as sharp as those in panchromatic image and spectral information is faithfully preserved without any obvious color distortion.

The results obtained from Brovey-transform, Gram-Schmidt methods and the proposed technology are reported in Figs. 3(c)-3(f), respectively. d) e) f) Fig. 3. Satellite images of WorldView-3: a) multichannel; b) NIR color composition; c) panchromatic; d) Brovey-transform; e) Gram-Schmidt methods; f) the proposed technology result. 4.2

Quantitative Analysis

As the visual analysis is very subjective and depends on the interpreter, a number of statistical analyses were performed [31-33]. To evaluate the spectral and spatial quality of pansharpened images we used relative dimensionless global error in synthesis (ERGAS) [ 11, 33 ]. Table 1 shows ERGASspectral obtained by known pansharpening methods (HSV, PCA, Gram-Shmidt, Wavelet) and synthesized images developed technology. It is clear, from its definition, that low ERGAS index values represent high image quality. One of the main difficulties is the quantitative assessment of visual image quality. To assess visual quality, an approach based on the calculation of information entropy is often used. Image entropy is a statistical feature that reflects the average information content in an image [16]. Fig. 4 shows a graphical representation of the value of information and entropy for original images and the synthesized multichannel image from our technology. The value of the synthesized image entropy value far exceeds the initial entropy multichannel image, this indicates that the new technology allows to improve information and detail objects multichannel images.

The correlation coefficient (CORR) is an important indicator reflecting the difference between the fused image and the original image [33]. This value ranges from -1 to 1. The best correspondences between fused and original image data show the highest correlation values. Table 2 shows the values CORR for the Gram-Schmidt, wavelet, packet wavelet, HSV, PCA, and new technology image fusion methods. So, Table 2 shows the best results for the proposed technology and the PCA method presents acceptable results. All other methods have a very low correlation values.

The SSIM shows the similarity with the original image. The structural similarity image quality paradigm is based on the assumption that the human visual system is highly adapted for extracting structural information from the scene [32]. Table 3 shows value of SSIM for the fused images in comparison with the multispectral image. All methods except of Wavelet and proposed technology are near zero, which confirms the fact that there is only slight similarity with the original image.

Quantitative analysis shows that existing methods for increasing spatial resolution lead to artifacts. The proposed technology increases the spatial resolution of multispectral aerospace images without color distortion. In this paper, we present the new pansharpening technology of high resolution satellite images. It is obtained, particularly, by utilizes the merit of the HSV fusion in smoothly integrating spatial resolution information and the merit of the wavelet fusion in preserving color information. The visual evaluation shows that the color of the fusion results of the proposed wavelet-HSV pansharpening technology is very close to the color of original MUL images for every data set; whereas the color of the standalone HSV fusion results and the wavelet fusion results are substantially distorted. Visual and quantitative analyses show that our presented technology preserves the original spectral features, can achieve high spectral and spatial quality and outperforms existing pansharpening methods.

Our further research will focus on improving the fusion accuracy of multichannel image fusion.

References

1. Hnatushenko , V.V. , Mozgovyi , D.K. , Vasyliev , V.V. , Kavats , O.O. : Satellite Monitoring of Consequences of Illegal Extraction of Amber in Ukraine . Scientific bulletin of National Mining University. - State Higher Educational Institution “National Mining University”, Dnipropetrovsk, № 2 ( 158 ). С. 99-105 , ( 2017 ).

2. Zhang, Y.: Problems in the fusion of commercial high-resolution satellite as well as Landsat 7 images and initial solutions . International Archives of Photogrammetry Remote Sensing and Spatial Information Sciences , 34 ( 4 ), p. 587 - 592 , ( 2012 ).

3. Hordiiuk , D.M. , Hnatushenko , V.V. : Neural network and local laplace filter methods applied to very high resolution remote sensing imagery in urban damage detection . 2017 IEEE International Young Scientists Forum on Applied Physics and Engineering (YSF) , ( 2017 ). doi: 10 .1109/ysf. 2017 . 8126648 .

4. Gnatushenko , V. : The use of geometrical methods in multispectral image processing . Journal of Automation and Information Sciences , Volume 35 ( 12 ), 1 - 8 , ( 2003 ). doi: 10 .1615/JAutomatInfScien.v35. i12 . 10 .

5. Kashtan , V. , Hnatushenko , V. , Shedlovska , Y. : Processing technology of multispectral remote sensing images . International Young Scientists Forum on Applied Physics 2017 , p. 355 - 358 . Lviv ( 2017 ). doi: 10 .1109/YSF. 2017 . 8126647 .

6. Meng , X. , Shen , H. , Li , H. , Zhang, L. , & Fu , R. : Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges . Information Fusion , 46 , 102 - 113 , ( 2019 ). doi: 10 .1016/j.inffus. 2018 . 05 .006 .

7. Vivone , G. , Alparone , L. , Chanussot , J. , Dalla Mura , M. , Garzelli , A. , Licciardi , G. A. , Restaino , R. , and Wald , L.: A critical comparison among pansharpening algorithms . IEEE Trans. Geosci . Remote Sens., vol. 53 , no. 5 , pp. 2565 - 2586 , ( 2015 ).

8. Ghassemian , H.: A review of remote sensing image fusion methods . Information Fusion , 32 , 75 - 89 , ( 2016 ). doi: 10 .1016/j.inffus. 2016 . 03 .003.

9. Xu , Q. , Li , B. , Zhang , Y. , Ding , L. : High-Fidelity Component Substitution Pansharpening by the Fitting of Substitution Data . EEE Transactions on Geoscience and Remote Sensing , vol. 52 , no. 11 , pp. 7380 - 7392 , ( 2014 ). doi: 10 .1109/TGRS. 2014 . 2311815 .

10. Sulaiman , A.G. , Elashmawi , W.H. , & El-Tawel , G.S.: A Robust Pan-Sharpening Scheme for Improving Resolution of Satellite Images in the Domain of the Nonsubsampled Shearlet Transform . Sensing and Imaging , ( 2019 ). 21 ( 1 ). doi:10.1007/s11220-019-0268-5.

11. Hnatushenko , V. , Hnatushenko , Vik., Kavats , O. , Shevchenko , V. : Pansharpening technology of high resolution multispectral and panchromatic satellite images . Scientific Bulletin of National Mining University, Issue 4 , 91 - 98 ( 2015 ).

12. Vivone , G. , Alparone , L. ; Chanussot , J.; Dalla Mura , M. ; Garzelli , A. ; Licciardi , G.A. ; Restaino , R. , Wald , L. : Acritical comparison among pansharpening algorithms . IEEE Trans. Geosci. Remote Sens . 53 , p. 2565 - 2586 ( 2015 ).

13. Aishwarya , N , Abirami, S. and Amutha , R.: Multifocus image fusion using Discrete Wavelet Transform and Sparse Representation . 2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET) , Chennai, 2016 , pp. 2377 - 2382 , ( 2016 ). doi: 10 .1109/WiSPNET. 2016 . 7566567 .

14. Rahmani , S , Strait, M , Merkurjev , D , Moeller, M , Wittman, T. : An adaptive IHS pansharpening method . IEEE Geosci. Remote Sens. Lett. , vol. 7 , no. 4 , pp. 746 - 750 ( 2010 ).

15. Haitao , Yin, Shutao, Li.: Pansharpening with multiscale normalized nonlocal means filter: a two-step approach . IEEE Transactions on Geoscience and Remote Sensing , vol. 53 , no. 10 , pp. 5734 - 5745 ( 2015 ).