Of-Nadir Angles Using Unsupervised Image Translation

Abstract

Synthetic aperture radar (SAR) images observed at diferent of-nadir angles have diferent intensities, and change detection methods using diference images do not work well. This problem hinders emergency response when there is no archive data with a consistent of-nadir angle as emergency SAR observation. In this paper, we investigate unsupervised image translation methods based on generative adversarial networks and autoencoders to detect flood and landslide areas using SAR images observed at diferent of-nadir angles. Comprehensive experiments of disaster detection using ALOS-2 PALSAR-2 images for three floods and two landslides show that the developed methods can significantly improve the accuracy of disaster detection using pre- and post-disaster images observed at diferent of-nadir angles. Autoencoders, disaster detection, generative adversarial networks, image translation, of-nadir angle, synthetic aperture radar in assessing the situation during emergency response. to assess flood extent assessment. ∗Corresponding author.

1. Introduction

Synthetic aperture radar (SAR) can acquire images in bad weather conditions and at night, and thus it is efective In particular, floods and landslides are often caused by heavy rainfall, and since the weather conditions are often bad, SAR images are more suitable than optical images for rapid detection of disaster areas in a wide area.

The straightforward method for detecting flood and landslide areas from SAR is image analysis [ 1 ], such as thresholding the diference between pre- and postdisaster images. This method assumes that the two images are observed at the same of-nadir angle. Recently, with the success of advanced machine learning algorithms, researchers have developed techniques using pairs of SAR data and deep convolutional neural networks [ 2, 3 ]. Although previous methods have excellent low the primary assumption of image analysis: having a set of pre- and post-disaster images with almost similar acquisition conditions (e.g., of-nadir angle). Attempts to perform change detection analysis using SAR images with diferent acquisition conditions have been focused CDCEO 2022: 2nd Workshop on Complex Data Challenges in Earth LGOBE

https://brunoadriano.com/ (B. Adriano); https://naotoyokoya.com/ (N. Yokoya)

Detection images caused by diferences in of-nadir angles, we investigate unsupervised image translation based on convolutional neural networks (CNNs). Encoder-decoder neural networks are applied to translate one image into an image obtained at the same of-nadir angle as the methods investigated in this paper: 1) image translation based on conditional generative adversarial networks (cGAN) [ 7, 8 ] that does not consider the presence or absence of change, and 2) image translation based on code-aligned autoencoders [ 9 ] that takes into account surface change. performance for detecting land changes, they still fol- To cope with nonlinear diferences between two period mainly on urban environments or forest areas by exploit- other image. The following subsections describe two 2.1. Conditional Generative Adversarial

Networks Figure 1 shows an overview of the cGAN-based image translation method that does not consider the presence or absence of change. The developed method consists of a generator with an encoder-decoder structure (i.e., U-Net [ 10 ]) to translate low-pass filtered image 1 into low-pass filtered image 2 during the training phase, and a discriminator to determine whether the transformed image is the true low-pass filtered image 2. The generator learns to trick the discriminator and the discriminator learns to detect that the translated image is false. In the inference stage, we discard the discriminator and apply the generator to the original image 1 to convert to image 2 with the same of-nadir angle. The method assumes that the low-pass filtered image pair contains relatively slight changes in the ground surface. Thus, diferences in images due to diferences in of-nadir angles are somewhat easier to transform using a U-Net than diferences in images due to changes in the ground surface; therefore, the generator only achieves the translation corresponding to the former. The diference image between translated image 1 and image 2 is subjected to thresholding to detect the change area.

2.2. Code-Aligned Autoencoders

We extend the image translation-based change detection method proposed by Luppino, et al. [ 9 ] that takes changes into account. Figure 2 shows an overview of the method. We train two diferent encoder-decoders for each disaster image pair by switching decoders. One is used to transform the pre-disaster image ∈ ℝ ℎ× to the post-disaster image ∈ ℝ ℎ× with the same of-nadir angle, and the other is used to transform the post-disaster ∈ ℝ ℎ× image to the pre-disaster image ∈ ℝ ℎ× with the same ofnadir angle. This dual encoder-decoder structure helps to increase the robustness of the final change detection. ℎ and are the height and width of the image used when training the CNNs. Each CNN consists of an encoder ( , ) that extracts semantic information about the ground surface and a decoder ( , ) that restores the style of the image (including the efect of of-nadir angles). The features ( , ) extracted for each image are passed to the other decoder to transform the image style. To achieve unsupervised learning, the final loss function is a combination of the four loss functions described below. The diferences from [ 9 ] are that the loss function is changed from the 2 norm to the 1 norm and that image modulation based on the Fourier transform is utilized for preprocessing.

Reconstruction loss: The solid black line process in Figure 2 indicates the image reconstruction; the reconstruction loss is the 1 norm of the diference between the inputs and outputs so that each encoder-decoder network acts as an autoencoder that outputs the same image as the one used for input: = [‖ −̃ ‖ 1] + [‖ −̃ ‖ 1], (1) where =̃ ( ( )) and =̃ ( ( )) .

Weighted translation loss: The following weighted translation loss is considered so that the result of image translation by switching decoders matches the other image in areas where there is no change in the ground surface: = [‖ ⊙ ( −̂ )‖

1] + [‖ ⊙ ( −̂ )‖ 1], (2) where =̂ ( ( )) , ̂ = ( ( )) , is a weight matrix that approaches 0 for pixels with ground surface changes and 1 for pixels without changes, and ⊙ is an element-wise product.

Cycle-consistency loss: Two iterations of an image translation should match the original image, which can be implemented by the following cycle consistency loss: = [‖ −̄ ‖ 1] + [‖ −̄ ‖ 1], (3) where =̄ ( ( )̂) and =̄ ( ( ) )̂ .

Code correlation loss: All three losses above represent the conditions that the translated image must satisfy as loss functions, but no explicit regularization is given as to what features the encoder should extract. We consider the following loss function so that the information about the change in correlation between all pixels in each input image is reflected in the features to be extracted by the encoder: defined by = , [‖ − ‖ where the correlation matrix ∈ ℝ × ( = ℎ × ) 1 √ , = 1 − ‖ − ‖2

, ∈ {1, ..., }, = [ ,1 , ..., , ], = [ ,1 , ..., , ], , = exp(−( − )2), , = exp(−( − )2). The correlation matrix ∈ ℝ × ( = ℎ × ) is defined by , = ( ) +

2

∈(1,...,) = max {‖ ‖, ‖ ‖}.

, ∈ {1, ..., }, (4) is (5) (6) Correlations between pixels of input images are assumed to reflect information about the ground surface rather than observational conditions such as of-nadir angles, because they are not afected by changes in the overall brightness of the image. So with we can restrict the features to have a similar correlation to force the encoder to output the features we want.

The final loss function is defined as the linear sum of the above four loss functions as follows: = + + + . The matrix is initialized to a 0 matrix in the initial stage of training, i.e., only , and are involved in the optimization. Along with updating the parameters of the CNNs, the weight matrix , which represents the presence or absence of change, is also updated according to the following equation: = 1 − , = ( , ̂) + (

, ̂ ) 2 , where denotes the calculation of the diference map by setting the threshold value. Partially cropped patches of the image to be analyzed are used to train the CNN models. After training, image translation is applied to the entire image, and Otsu’s method [ 11 ] is applied to the diference image for final change detection. 2.3. Image Modulation Based on Fourier

Transform

The success of the CAA algorithm depends on whether or not it can proceed from a reasonable weight matrix in the initial stage of learning. If the diferences in of-nadir angles are large, there is a problem of not converging to a good local optimal solution. Therefore, as a pre-processing step, we use as the input image for the Image 1

(Predisaster) Image 2

(Postdisaster) ilt u d e A m ilt u d e x + x (7) the amplitude of the low-frequency component of image Image 1 2014.10.6

Modulated ! = 5

Modulated ! = 10

Modulated ! = 20

Modulated ! = 40

Image 2 2015.9.12 ulation for Kinugawa (9.12) data. (Best viewed with zoom CAA algorithm an image in which the low-frequency component of one image (pre-disaster image) is made closer to the low-frequency component of the other image (post-disaster image), thereby reducing the diference between the two images in terms of overall intensity. 2. The simplest method for replacing the amplitude is to use a two-dimensional rectangular function to replace components below a certain frequency, but in this paper, a linear summation using a two-dimensional Gaussian function was employed for more natural image modulation. The standard deviation ( ) of the Gaussian function is a parameter that adjusts the degree to which highfrequency components are modulated. Finally, the modulated image is obtained by applying a two-dimensional inverse Fourier transform to the phase of image 1 and the amplitude modulated by the linear sum. Figure 4 shows an example of image modulation based on the Fourier transform. It can be seen that after modulation, the brightness of the entire image approaches that of image 2, but the detailed pattern of the image is inherited from image 1.

3.2. Evaluation Method

3. Experiments

3.1. Data The F-score is used as a metric to evaluate the accuracy of change detection. The efectiveness of the investigated methods is verified by comparing the following ifve methods.

Experiments on disaster area detection are conducted using ALOS-2 PALSAR-2 imagery of three floods (the September 2015 Kanto/Tohoku heavy rainfall, the July • Thresholding of diference images of the same 2018 heavy rainfall, and the July 2020 heavy rainfall) and of-nadir angle images; it will be referred to as two landslide disasters (the July 2018 heavy rainfall and the 2018 Hokkaido Eastern Iburi earthquake) in Japan. Reference Binary polygon data of inundation areas and landslide • Thresholding of diference images of diferent ofdisaster areas published by the Geospatial Information nadir angle images Authority of Japan (GSI) are used as the ground truth data. • cGAN-based image translation For the September 2015 Kanto/Tohoku heavy rainfall, • CAA-based image translation there are images observed on three diferent days after • CAA-based image translation using image moduthe disaster and the corresponding ground truth data, lation based on Fourier transforms as preprocessso they are treated as three independent experiments. ing (denoted as CAA-FT) Therefore, there are seven experimental datasets in total. In order to evaluate the efectiveness of image translation

Table 1 shows the observation conditions and image for change detection, the change detection method in this sizes for the seven datasets composed of 19 ALOS-2 study was unified into the simplest thresholding method. PALSAR-2 images used in this study. For those datasets Thresholding is applied to the case of reduced pixel values having 3 images, we have a pair of images with the same in the case of floods, and to both increased and decreased of-nadir angle before and after the disaster. The results pixel values in the case of landslides. of change detection using the same of-nadir angle images are used as reference information to evaluate the accuracy of the developed methods. In all experiments, 3.3. Results images with speckle noise reduced by bilateral filtering were used as input. detection in the experiments using the Kinugawa (9.12) 3.3.1. Flood detection and Saga datasets, respectively. Compared to thresholding the diference images of diferent of-nadir angle images, the investigated methods provide superior detection results in the six experiments. More detailed results and discussion on flood detection and landslide detection are presented below.

For flood detection, the F-scores of around 0.35 to 0.65 were achieved, which can be considered to be of practical accuracy considering that the simplest change detection method of thresholding is used. For the Kurashiki and

Kinugawa (9.12) datasets, the F-scores are comparable to Pre-disaster 32.8 degree Post-disaster 50.5 degree

Thresholding (same off-nadir angle) that of using the same of-nadir angle images, indicating Table 3 that our objective is achieved. For the Saga dataset, the F- Of-nadir angle conditions and the efectiveness of the methscores of the investigated methods exceeded the reference ods under investigation. One, two, or three • signs mean low, value, which may be due to the fact that the seasonal mideum, and high. diference for the same of-nadir angle images is larger Angle dif. cGAN CAA CAA-FT than that for diferent of-nadir angles. A comparison of the performance among the methods under investigation 5–10° ••• •• • in this paper reveals the following trends. 10–25° • •• ••• • When the diference in of-nadir angles is about 5°to 10°, as in Kurashiki and Kinugawa (9.12), the cGAN-based image translation that does not consider the presence of change is efective. • When the diference of of-nadir angles is about 10°to 25°, as in Kinugawa (9.13), Kinugawa (9.16), and Saga, the CAA-based image translation focusing on the unchanged area is efective. • If the diference in of-nadir angles is not large, image modulation based on the Fourier transform is not necessary for CAA.

achieve suficient detection accuracy even when using the same of-nadir angle images. In [ 12 ], F-scores of 0.33 and 0.58 were achieved for Hiroshima and Hokkaido, respectively, using an algorithm specialized for landslides detection. In this paper, the simplest thresholding was used as a basis to evaluate the efectiveness of change detection based on image translation. However, it is expected that the combination of landslide detection algorithms that take land cover and topography into account and the image translation methods investigated in this paper will improve the detection accuracy of landslide areas to a practical level.

4. Conclusion and Future Outlook

The above trends are consistent with the conditions and design principles assumed by each method, suggesting that reasonable experimental results were obtained.

The cGAN-based method assumes that image diferences due to diferences in of-nadir angles are relatively easier to transform using a CNN than image diferences due to changes in the ground surface, and this condition is considered to be satisfied when the diferences in ofnadir angles are relatively small. If this assumption is not satisfied, i.e., the diferences in images due to diferences in of-nadir angles are relatively large, then the CAAbased methods, which perform image translation while taking into account the presence or absence of changes in the ground surface, is efective. Furthermore, it was shown experimentally that the pre-processing based on the Fourier transform contributed to the improvement of accuracy by reducing the dificulty of the problem.

We studied the potential of image translation via unsupervised deep learning to detect flood and landslide areas using SAR images with diferent of-nadir angles. We investigated a cGAN-based method that does not consider changes and a CAA-based method that focuses on unchanged areas of the ground surface during image translation. Experiments on seven cases of flood and landslides disasters demonstrate the efectiveness of the development methods.

Table 3 summarizes the relationship between of-nadir angle conditions and the efectiveness of the methods under investigation for flood detection. When the diference 3.3.2. Landslide detection in of-nadir angle is about 5°to 10°, cGAN-based image translation without considering change is efective, and In the experiment for detecting landslides using the Hi- when the diference in of-nadir angle is about 10°to 25°, roshima dataset, the CAA-based method showed a clear CAA-based image translation focusing unchanged areas improvement in accuracy. In the Hokkaido experiment, can significantly improve the accuracy of flood detection. the thresholding results for the diference images of dif- Cases with larger diferences in of-nadir angles (greater ferent of-nadir angle images were almost the same as the than 25°) are not included in the experiments and require reference value using the same of-nadir angle images, further verification. and no further improvement could be obtained by the The overall accuracy in detecting landslide areas was investigated methods. This may be due to the fact that low because the simplest method of change detection, the diference in of-nadir angles is so small (4.5°) that thresholding of diference images, was used. Combinits efect on the observed image is negligible. Although ing landslide detection algorithms that take land cover the presented methods did not improve the accuracy of and topography into account with image translation is a change detection, this is a reasonable result. future challenge.

In the experiment for detecting landslides, the F-score was generally low. This is because the detection of landslide areas is more dificult than the detection of inundation areas, and the simple thresholding process cannot

Acknowledgments

The authors would like to thank the Japan Aerospace Exploration Agency for providing ALOS-2 PALSAR-2 data.

References

[1] L. Landuyt , A. Van Wesemael , G. J. Schumann , R. Hostache , N. E. Verhoest , F. M. Van Coillie , Flood Mapping Based on Synthetic Aperture Radar: An Assessment of Established Approaches, IEEE Transactions on Geoscience and Remote Sensing 57 ( 2019 ) 722 - 739 . doi:1 0 . 1 1 0 9 / T G R S . 2 0 1 8 . 2 8 6 0 0 5 4 .

[2] Y. Li , S. Martinis , M. Wieland , Urban flood mapping with an active self-learning convolutional neural network based on TerraSAR-X intensity and interferometric coherence , ISPRS Journal of Photogrammetry and Remote Sensing 152 ( 2019 ) 178 - 191 . doi:1 0 . 1 0 1 6 / j . i s p r s j p r s . 2 0 1 9 . 0 4 . 0 1 4 .

[3] X. Li , Z. Du , Y. Huang , Z. Tan , A deep translation (gan) based change detection network for optical and sar remote sensing images , ISPRS Journal of Photogrammetry and Remote Sensing 179 ( 2021 ) 14 - 34 . URL: https://www.sciencedirect.com/ science/article/pii/S0924271621001842. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . i s p r s j p r s . 2 0 2 1 . 0 7 . 0 0 7 .

[4] P. Ge , H. Gokon, , K. Meguro , Building damage assessment using intensity sar data with diferent incidence angles and longtime interval , Journal of Disaster Research 14 ( 2019 ) 456 - 465 . doi:1 0 . 2 0 9 6 5 / j d r . 2 0 1 9 . p 0 4 5 6 .

[5] J. Tao , S. Auer , P. Reinartz , R. Bamler , Object-based change detection for individual buildings in sar images captured with diferent incidence angles , 2013 IEEE International Geoscience and Remote Sensing Symposium - IGARSS ( 2013 ) 1238 - 1241 .

[6] M. W. Lang , P. A. Townsend , E. S. Kasischke , Influence of incidence angle on detecting lfooded forests using c-hh synthetic aperture radar data , Remote Sensing of Environment 112 ( 2008 ) 3898 - 3907 . URL: https://www.sciencedirect.com/ science/article/pii/S0034425708002228. doi:h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . r s e . 2 0 0 8 . 0 6 . 0 1 3 .

[7] I. Goodfellow , J. Pouget-Abadie , M. Mirza , B. Xu , D. Warde-Farley , S. Ozair , A. Courville , Y. Bengio , Generative adversarial nets, Advances in neural information processing systems 27 ( 2014 ).

[8] P. Isola , J.-Y. Zhu, T. Zhou , A. A. Efros , Imageto-image translation with conditional adversarial networks , in: Proceedings of the IEEE conference on computer vision and pattern recognition , 2017 , pp. 1125 - 1134 .

[9] L. T. Luppino , M. A. Hansen , M. Kampfmeyer , F. M. Bianchi , G. Moser, R. Jenssen , S. N. Anfinsen , Codealigned autoencoders for unsupervised change detection in multimodal remote sensing images , IEEE Transactions on Neural Networks and Learning Systems ( 2022 ).

[10] O. Ronneberger , P. Fischer , T. Brox , U-net: Convolutional networks for biomedical image segmentation , in: International Conference on Medical image computing and computer-assisted intervention , Springer, 2015 , pp. 234 - 241 .

[11] N. Otsu , A threshold selection method from graylevel histograms , IEEE transactions on systems, man, and cybernetics 9 ( 1979 ) 62 - 66 .

[12] B. Adriano , N. Yokoya , H. Miura , M. Matsuoka , S. Koshimura , A semiautomatic pixel-object method for detecting landslides using multitemporal alos-2 intensity images , Remote Sensing 12 ( 2020 ) 561 .