The implementation of deep learning (DL) models has significantly improved the accuracy and automation of remote sensing (RS) image classification tasks, such as landslide detection. The reason is that DL models have independent feature learning and strong computing capabilities and have attracted continuous attention in modifications and enhancements through numerous model-centric eforts. In practice, however, the impact of the quality of training samples on classification performance is usually ignored. This study uses a model-centric approach in which a U-Net network is regarded as a baseline model. A ResNet-34 model is used to optimize the baseline model, and the optimized model is further enhanced by adding an attention mechanism. However, in the data-centric approach, the baseline model is only trained based on the enhanced training samples. Our data-centric approach increased the F1-score by over 13 percentage points, which is the same increase as the most sophisticated and complex model-centric approach.
As remote sensing (RS) imagery has become the basis of data collection across various fields, such as agriculture, environment, and disaster risk management, critical information can be extracted from such multi-temporal and multi-resolution images through image classification, one of the most critical aspects of such data processing is selecting the appropriate method to use. For many years, the remote sensing community has used artificial neural networks (ANN) such as Multi-Perceptron Layer (MLP) as a conventional method of image classification [ 2]. Until and DenseNet [4].
In RS tasks such as image classification, however, the goal usually is to label every pixel within an image, and DL semantic segmentation techniques such as Fully Convolutional Networks (FCN) are used to achieve that goal [5, 6]. The U-Net [7] algorithm, which utilizes encoderdecoder architectures to improve FCN, is widely used by tection, although it was initially designed for medical image segmentation. Other region-based models are used to detect and segment objects, including Faster R-CNN and Mask R-CNN, successfully applied to landslide inventory mapping [8]. These segmentation models have also beobject detection, and time series analysis [1]. However, the RS community for image segmentation and object deels like support vector machines and ensemble classifiers recently, however, conventional machine learning mod- come more sophisticated and advanced by incorporating concepts like attention mechanisms and or incorporatsuch as random forests nearly replaced ANN models for ing backbone models and weights such as the residual provide acceptable performance even with limited la- such the way of working of human vision. tasks like image classification and change detection because they can handle data with high dimensions and beled data [2]. Recent developments in computer vision and graphics processing units (GPUs) have led to a rise in the popularity of deep neural networks within the RS community for diferent tasks, as they have generated robust results in image classification and image segmentation [3]. Several image classification models have been nEvelop-O integrating OBIA with U-net resulted in an F1 score value nearly 8% higher than the baseline ResU-Net model for the landslide detection task. In another study, Donget tection by adding a multi-scale feature-fusion module, developed, such as AlexNet, VGG net, GoogleNet, ResNet, object-based image analysis (OBIA) for landslide detec© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License al. [12], improved the U-Net’s ability for landslide dea residual attention network, and a data-dependent up sampling method. Their enhanced network, named LUnet could increase the F1 score by more than 3%. While Ghorbanzadehet al. [13] followed a data-centric strategy by preparing six training data sets based on optical data and diferent topographic information to evaluate the performance of a deep-learning convolution neural network (CNN) for landslide detection, a study area in Nepal. Their most remarkable improvement increased the mIOU by more than 17 percentage points. Yanget al. [14] have done a training samples enhancement and developed a background-enhancement technique that could support distinguishing landslides and similar background features for training the Mask R-CNN model. The F1 score was significantly higher (22.38%) than the one obtained using only satellite images as input data in their experiment.
As a result of a literature review on applying DL to RS applications, including landslide detection, the focus is mainly on model-centric approaches such as adapting, comparing architectures, and developing advanced models. At the same time, input data plays a less important role in model performance. In the model-centric approach, the input data is the same, while the main efort is focused on code and developing experimental research to improve the model performance. This involves selecting the best model architecture and training process from various possibilities [15]. On the other hand, in the data-centric approach, the goal is to systematically alter, synthesize, and improve datasets to increase the model’s accuracy with a fixed architecture [ 16]. To our best knowledge, no study has comprehensively compared or discussed model-centric vs. data-centric approaches in landslide detection purposes. Since most DL studies for RS tasks are model-centric, in this experimental case study, we aim to compare and evaluate the performance of model-centric and data-centric approaches in landslide detection using Sentinel-2 imagery, ALOS elevation data U-net segmentation method.
A magnitude 6.6 earthquake struck Eastern Iburi, Hokkaido, Japan, on September 6, 2018. There were extensive damages caused by this incident, including power cuts, damage to transmission and distribution networks, and damage to the Tomato-Atsuma Power Station, which supplies electricity to Hokkaido Island. There were 41 fatalities in all, 36 of which were caused by landslides triggered by the earthquake. Typhoon Jebi brought torrential rains to the region just a day before earthquake, which made hills unstable and prone to landslides. This caused nearly 5600 landslides in the area. The result was a significant number of shallow landslides as well as a Japan
Hokkaido 0 125 250 50K0ilometers e t i S g n i n i a r T e t i S g n i t s e
T Landslides few planar and spoon-type deep landslides. A landslide inventory map was generated and updated by the Geographical Survey Institute (GSI) of Japan using aerial orthophotos, very high-resolution aerial images, and a 10m resolution digital elevation model (DEM) [9].
As part of this study, an inventory map (around 4950 landslides in an area of 43 km2) in shape file format provided by GIS in ESRI was acquired and used as ground truth for further analysis. Landslides were detected using Sentinel-2 multispectral imagery in this case. We applied atmospheric corrections to images using Sen2Cor [17], a SNAP plugin. We generated slope layers using the 12-m ALOS DEM, which is an important data set for mapping landslides [3]. Sentinel 2 images range in spatial resolution from 10 to 60 meters, so we selected only bands 2-4 and 8, which have a 10-meter resolution and excluded other bands. The generated slope layer was re sampled to 10-m and stacked with the Sentinel bands. Figure 1 shows the study area and landslide inventory. 3. Methodology 3.1. Model-centric approach 3.1.1. U-Net associated with the highest weights. A detailed description of the attention mechanism in CNN-based models for the RS applications is provided by [1]. In this case, an attention mechanism is added to the U-Net model with the ResNet 34 backbone for landslide detection.
As stated in the introduction section, U-Net was initially introduced for the segmentation of biomedical images 3.2. Data-centric approach Ronnebergeret al. [7]. Because of its robust performance, it has been used for a wide range of image segmentation In this approach, the U-Net architecture applied by [9], problems, including remote sensing image classification which was introduced as our baseline model is used withand object detection [9]. In U-Net Architecture, encoder out any model enhancement. However, Data enhanceand decoder are the two main components. By apply- ment will be done by synthesizing and augmenting training convolution, activation functions, and pooling oper- ing samples. Therefore, along with the 10-m Sentinel 2 ations, the encoder learns how to abstractly represent images and slope layer, normalized diference vegetation the input image. Pooling reduces computational cost in index (NDVI) will be generated and fed into the model this part, but spatial data is lost [7]. Through operations as well. This index is helpful for discriminating some such as transpose convolution model, the decoder part of landslides that removed the surface vegetation from attempts to restore the original size of the abstracted the background objects [14, 13]. Moreover, using the representation. Concatenating the output of transpose OBIA concept and multi-resolution segmentation (MRS), convolution with the skip connection feature map in the which is a bottom-up segmentation technique based on encoder part is the skip-connection feature map at the the pairwise region-merging approach the size of the same level as the output of transpose convolution. While generated object is controlled by the scale factor [19]. in the encoder part the number of feature channels is In order to avoid errors such as over-segmentation and doubled at each down-sampling, in the decoder part it under-segmentation an index called as object fitness inis diminished by half until in the final layer a convolu- dex (OFI) introduced by [20] is applied to guaranty the tion (1x1) is used (in this case with sigmoid activation) to quality of objects. The mean values of objects for each map the channel patterns to a given number of classes. image band, including NDVI and slope, will be calculated, The architecture used by Ghorbanzadehet al. [9] will be and then exported in a raster format to be stacked with implemented for this study. another dataset as input to the U-Net model.
ResNet serves as a backbone for many computer vision tasks, including remote sensing image classification and segmentation [9]. In 2015, it won the ImageNet challenge with extremely deep neural networks (more than 150 layers). ResNet’s success is mainly due to its novel architecture that introduced skip connections for the first time, which add the output from the previous layer to the layer ahead. This alternative shortcut path allows gradient to flow through and prevents the problem of vanishing gradient. The baseline model of the U-Net design is used in this research, but with ResNet-34 acting as a backbone.
Bahdanauet al. [18] introduced the attention mechanism to enhance the performance of the encoder-decoder models for machine translation system. Later its variants were used in other application including the RS applications. Through a weighted combination of encoded input data, the decoder has access to the most valuable parts of the input sequence, thus, the most relevant parts will be
In this study, a tile size of 128 × 128 without any overlap was used as an input to all applied models. The accuracy of the model was also validated by selecting 30% of training data sets at random. As our task is binary classiifcation, we applied Sigmoid as the activation function in the last layers and rectified linear activation function (ReLU) in the earlier layers. All models were set at 100 epochs, but a function was defined to save the model on the epoch number to ensure minimum losses. Our DL models were all implemented in Python using TensorFlow API and Keras library, for segmentation part we used eCognition software. As previously noted, data augmentation was only applied to a data-centric approach.
Each model was evaluated based on standard accuracy assessment metrics of the precision, recall and F1-score. A loss and F1-score of 0.14 and 0.88 were achieved during the training of the conventional U-Net model while, these values were 0.26 and 0.68 for validation data. A total of almost 4 million parameters were trained in this model. For test area, the trained model was used to detect landslides; the accuracy assessment results showed precision, recall, and F1-score values of 0.76, 0.48, and U-Net
0.59, respectively. The same U-Net was trained in the following, but with ResNet 34 as a backbone model. The total number of parameters for this model was 23 million, but only 1.8 million parameters were trained, and for the rest pre-trained weights were applied. Training Loss and F1-score values for U-Net with ResNet 34 backbone were 0.13 and 0.93, respectively, and for the validation data set, the scores were 0.18 and 0.90. Although the model’s accuracy on training data and validation is quite close, its performance in the test area did not provide much higher accuracy. Precision, recall, and F1-score for U-Net with ResNet backbone were 0.67, 0.72, and 0.70, respectively. Finally, the most complex version of U-Net that includes both ResNet backbone and attention mechanism with 10.5 million parameters was trained. For training, loss and F1-score values were 0.05 and 0.95, and for validation were 0.08 and 91, respectively. However, like U-Net with ResNet backbone model, the performance of the model in the test area even with adding attention mechanism to the model was not significant, and values 0.88, 0.62, 0.72 were achieved for precision, recall, and F1-score, accordingly. It provided the best precision score among other models.
For the data-centric approach, only the base line U-Net model with the same architecture was used as for the first scenario in the model-centric approach. However, the input data has been modified. The OBIA features were stacked with other images, and then data augmentation techniques such as flipping (horizontally and vertically) and rotating (90, 180, 270 degrees) were performed. This resulted in 10395 image patches being fed to the network instead of 2079 image patches. In addition, data augmentation was not applied to validation data. The U-Net model achieved values of 0.13 for loss and 0.93 for F1-score during training. With validation data, however, the values were 0.18 and 0.90. The trained model was used to predict test data, resulting in precision, recall, and F1-score of 0.71, 0.73, and 0.72. The figures 2 and 3 are respectively depicting the training curve of a model and the landslide prediction map for a model-centric and a data-centric approach to landslide prediction.
In the model-centric approach, using the same labeled data while varying architecture and parameters resulted in varying accuracies in training and validation phases, general U-Net models performed relatively poorly (underfitting error) in the training process compared to others. U-Net based on ResNet Backbone, however, performs better during training, while the best performance is achieved with U-Net based on ResNet and attention mechanism. Loss values for both training and validation data indicate that by increasing models’ parameters higher accuracy during training can be achieved. But getting such high accuracy throughout training can be a sign of overfitting. Therefore, prediction results by such a model are evaluated with inventory data. Consequently, excepting the general U-Net provided the lowest accuracy with a recall of 0.48 and an F1-score of 0.60. which means it U-Net was able to detect only 48% of landslides. Furthermore, U-Net with ResNet backbone provided much better performance compared to the general U-Net model, with a recall of 0.72 and F1-score of 072. Finally in model-centric, the best performance according µ µ
U-Net ResNet(34)U-Net Training the model using synthesized and augmented data provided a great performance on both training data and validation. And evaluating predicted landslide with inventory map indicated the F1-score of 0.72 while great consistency between other metrics such as precision with 0.71 and recall of 0.73. This experiment clearly shows that by generating/synthesizing data and argumentation available data higher accuracy can be achieved even with simple model architecture. For example, the performance and accuracy of the data-centric U-Net model were quite similar to an advanced U-Net model with a ResNet backbone and attention mechanism, and in terms of F1-score, the diference in both models’ performance was 1%.
µ to F1-score achieved by the U-Net model based on ResNet backbone and attention mechanism, the score achieved was 0.73 although the recall value was 0.62 it provided the highest precision of 0.88. In Data-Centric, the conventional U-Net with the same architecture was used as the fixed model, while data went through argumentation. the impact of the data-centric approach on the model transferability to new areas is the focus of our next work.
This research was funded by the Institute of Advanced Research in Artificial Intelligence (IARAI) GmbH, Vienna, Austria.