- Accurate detection of the cloud in remote sensing analytics is considered an essential task in remote sensing imagery with various spectral, temporal and spatial information. As a result we propose a multifeature fusion deep convolutional neural networks for analysis of remote sensing satellite images so as to detect clouds, which are the region of interest. To ensure the algorithm was trained with data acquired from multiple satellites, the Landsat 7 ETM, Landsat 8 OLI/TIRS and Gaofen-1 wide field view datasets were used. The experimental results obtained showed that the proposed model gave accuracy, precision and recall measures of 95.2%, 89% and 89.9%respectively. The developed algorithm posted consistent and accurate results for cloud detection using satellite images that had clouds of different types and those obtained over different land surfaces that contain other objects in the images.
Presence of clouds is seen to pose a challenge in the process of extracting information of the surface and/or atmosphere in remote sensing satellites in addition to affecting the amount of radiation a surface can receive[1]. Accurate detection of the cloud in remote sensing analytics has so far been seen to be a challenging task due to the various shapes the clouds may take in addition to having different ground objects captured in the satellite images[2]. To identify clouds given a satellite image, three approaches have so far been evaluated, that is, threshold based approaches which evaluate the reflectance and brightness across a given channel so as to detect presence of clouds, machine learning algorithms such as support vector machine(SVM)[3], artificial neural networks (ANN)[4] and random forest[5] that learn from handcrafted features so as to detect presence of clouds in an image and deep-learning technique[6] that automatically learn high complex features given a training dataset.
[7]reviewed literature on cloud detection using remote sensing satellite imagery from 2014 -2018.In their work, it was reported that most researchers explored a variety of cloud detection forms such as Cloud/No Cloud, Thin Cloud/ Thick Cloud and Snow/Cloud using threshold based techniques, deep learning algorithms and machine learning algorithms such as ANN, decision trees, random forests and Bayesian classification. Threshold based techniques were observed to perform differently for different climatic conditions or areas with different surface type thus making them have poor universality while machine learning algorithms that used handcrafted features were observed to be dependent on the feature selection process as different people have different understanding of clouds and their features. As a result, deep learning algorithms became more appealing due to their capability of automatically extracting highly complex features based on the spectral, temporal and spatial information provided in the training dataset.
Depending on the task being performed, deep learning techniques for cloud detection have been categorized to three major categories that is, patch based deep learning approach which uses an image patch as an input and gives a labelled output indicating whether the image is cloudy or not cloudy, region based deep learning techniques that segment an input image to different regions that are labelled using a pretrained network and a pixel level deep learning technique that takes a fixed size input image and trains the model to output a pixel level labels that have the same size as the input image.
In this work, a multi-feature fusion deep convolutional neural network (MFF – DCNN) to predict presence of cloud given a remote sensing satellite image as an input is proposed. A single set feature, created by fusing the spectral, temporal and spatial features, is used to train the developed model for cloud detection.
II . RELATED WORK [8] Developed a CNN algorithm for cloud detection based on a residual network (ReSNet) architecture which was crafted from the u-net architecture by adding a clipping layer, batch normalization and halving the depth of feature maps. The dataset obtained from the NASA Landsat 8 satellite consisted of five spectral band combinations that is the red/green/blue/infrared (RGBI) band, the red/green/blue (RGB) and the green band alone. In their work, they noted that the develop CNN model posted good results for semantic segmentation in addition to improving the performance levels and reducing the time it took to train the model by reducing the requirements during the preprocessing phase. To improve on the performance measure of the CNN algorithm for cloud detection, they stated the need for a method that would fully incorporate the spectral, spatial and temporal dimensions.
[9] Proposed a technique of detecting clouds that was based on cloud segmentation by fusing multi-scale convolutional features (MSCF) with aim of improving on the accuracy of the convolutional neural network (CNN) for object detection especially when using a multispectral image that contains the visible and infrared bands only. The proposed deep learning technique was based on fully convolutional network (FCN) for pixel-to-pixel semantic segmentation and Semantic Segmentation (SegNet) architecture which was built as a convolutional encoderdecoder for semantic segmentation of the pixel values. To train and evaluate the proposed model, images from Gaofen-1 WFV satellite were used and the performance posted by the developed model compared with two other techniques, that is, a multi-feature combined method (MFC) and a deep convolutional network (DCN).The obtained results shows that the MSCN model performs better with an accuracy level of 97.85% compared to the MFC model that posted an accuracy of 96.80%. Additionally, the MSCF was seen to keep details of the cloud boundaries on the cloud mask produced. For future studies, they recommended use of cloud images obtained from different satellite imageries to investigate if their suggested model can be generalizable on the different datasets.
[10]proposed a two-step deep learning technique for
cloud detection on remote sensing satellite imagery. The first
process involved use of a feature concatenation network
which would obtain the cloud probability map from deep the
convolutional neural network while the second process
involved extraction of multilevel structural features using a
multi-window guided filtering so as to refine the cloud mask.
To validate the proposed model, the 502 Gaefen-1 WFV
cloud images collected from May 2013
To train and validate the model, satellite images obtained from different satellites and covering different land surfaces as illustrated in the fig 1 were used. A total of 90 Landsat 8 Operational Land Imager/Thermal Infrared Sensor (OLI/TIRS) satellite images with a resolution of 30m which are provided to the public by [14], 160 Landsat 7 Enhanced Thematic Mapper Plus(ETM+) with a resolution of 30m[15] and 100 Gaofen-1 wide field view(WFW) with a resolution of 16m[16] were used. The Landsat-8 images were broadly categorized to eight biomes, that is, barren, forest, grass/crop, shrub land, urban, water and are further divided to four classes, that is, cloud, thin cloud, cloud shadow and clear. According to [17], data used to train a model should be more as compared to training dataset with a preferred percentage split of 70/30 whereby 70% of the total dataset is used to train the model while the other 30% is used to evaluate its performance. In our study, to obtain the training and test dataset, percentage split was used to randomly split the datasets whereby 70% of the each dataset was used for training the model while the remaining 30% was used for testing the performance of the developed model for cloud detection given satellite images.
The proposed MFF – DCNN for cloud detection is composed of a deep coarse network for extraction of high level features and three deep fine extraction network for separation of the cloud pixels from other objects present in an image as illustrated in fig 2.A fully connected (FC) layer is used to flatten the outputs obtained from the coarse and the fine modules and the output obtained from this layer fed to a feature fusion layer that is used to fuse features obtained from the four network components. The output from the feature fusion layer used for classification.
1) Deep Coarse Network: The deep coarse network (DCN) is constituted of three convolutional layers for the purpose of extracting high level features. The first layer is structured such that it is comprised of 36 filters and a kernel size of 3*3. The rectified linear unit(ReLU) activation function is then be applied to the convolved patches and a 2*2/2 max-pool function applied to each response generated. The second and the third convolution layers are then modelled such that they are made of 64 filters with the ReLU activation function and a 5*5 kernel and a 2*2 max-pooling with stride 2.
Fig 1: Display of the Obtained Datasets on various land surfaces Source: [18]
images acquired from remote sensing satellites are rarely annotated and also lack bounding boxes to represent the most likely region of interest (ROI)[19]. As a result, this deep fine extraction network is developed to help in identification of the ROIs. The three deep fine extraction layer are built using the ResNet50 architecture which has so far proved to have better performance for object detection as compared to the other convolutional architectures in addition to helping to mitigate the vanishing gradient problem[20]. The areas marked by the bounding boxes are then extracted using ROI pooling layer so as to come up with a feature map to be fed to a fully connected layer that would compute the score for the input image for each class. The fig 3 illustrates the structure of the deep fine extraction module.
3) Feature Fusion layer: The feature fusion layer is introduced to the architecture so as to ensure that the features extracted from the deep-coarse and deep-fine networks are combined to form a single feature set for classification.To avoid overfitting and improve on the generalization of the model, cross entropy loss function[21] was used to regularize the feature fusion process and the softmax regression function[22], defined in equation (1) used for classification.
( ) [ ( ) ] ( )
(1) where x is the filter i score obtained from the previous layer and is the corresponding output. Fig 2: The Proposed Multi-Feature Fusion Deep Convolutional Neural Network for cloud detection Fig 3: The Architecture of the Deep Fine Extraction Network
The developed model was trained to detect clouds given a remote satellite image dataset that captures different land surfaces in addition to containing spatial, temporal and spectral information. To evaluate the performance levels for extracting and fusing multiple features proposed model, for cloud detection
using the the accuracy, precision and recall measures as
specified (4)respectively were used.
in equation (2),(3) and (2) (3) (4) Where TP is the true positives, TN is the true negatives, FP is the false positives and FN is the false negatives.
In our work the performance measure of the FCN network proposed by [9], multi-feature fusion of point and block feature using SVM classifier proposed by [2] and a multilevel feature cloud detection FCN proposed by [10] are compared with our proposed model. These models were evaluated using the test datasets that contained a total 57 Landsat-8 images that were different from the images used to train the model and the
accuracy results obtained are as summarizer in table 1.These results showed that our model performed the best with an accuracy level of 99.06 which was 1.21% higher than that of the fully convolutional network implemented using the Segnet architecture.
Which performed as the second best model for cloud detection. Of the four compared models, the model composed of the feature concatenation and window guided filtering gave the lowest accuracy level of 92.92%. Furthermore, the recall and precision measure obtained when testing our model were 89.9% and 89% respectively and an increase in the number of epochs to about 25 epochs was seen to lead to an increase in both the training and validation accuracy of the proposed model but this was seen to stabilize. Further increase of the number of epochs led to an unstable increase and decrease in the training and validation accuracy as illustrated in the fig 4.
The ability of the proposed MFF – DCNN to effectively detect both thick and thin clouds is mainly been attributed to the use of
ResNet50 skip connections in the DFCN that enables the model utilize all available feature for cloud detection. Fig 4:
Proposed Model training and validation accuracy against the number of epochs
According to[20] the residual block of the ResNet architecture as illustrated in the fig 4 enables a model train deeper neural networks by optimally tuning the number of layers while training the model.
Consequently, it is associated with its high capabilities of addressing the vanishing gradient problem that is frequent especially when more layers are added to a deep learning model. Fusing both the high level features obtained from the deep-coarse network and the low level features from the DFCN enables the model take into consideration all features available in remote sensing satellite imagery, that is, the spectral, textural and spatial information during the training process. Thus the model is capable of learning more information and as a result enable it improve on its predictive predictability.
Fig 5: ResNet Residual Block
V. CONCLUSION
In this work, we propose a multi-feature fusion extraction based on deep convolutional neural network for cloud detection in remote sensing analytics given a satellite image that consists of spectral, spatial and textural information. The proposed MFF – DCNN model architecture consisted of a deep-coarse network for extraction of high level features and deep-fine network for extraction of low level features. For identification of the region of interest, feature fusion layer made of a fully connected layer was then used to fuse features identified in the deep-coarse and deepfine networks and the result fed to a softmax regression function for classification. The cross entropy loss function was then used to regularize the outputs. The quantitative results obtained showed that the proposed model was capable of performing well given datasets that have different clouds types with varying cloud size and density and as a result, it was concluded that this model can also be replicated to different scenarios for accurate and reliable cloud detection. The proposed model is thus seen to be useful in providing insights about clouds in remote sensing analytics and as a results it has proved to be useful in tasks such as prediction of the amount of solar irradiance a given surface can receive given the locations’ atmospheric and cloud conditions.
For future works, we recommend evaluation of the proposed model on images obtained from other satellites such as Sentinel-2, SPOT-5 and the Moderate Resolution Imaging Spectroradiometer (MODIS) data so as to evaluate generalizability of the proposed model on different datasets. Additionally, more research should be made on how to improve on the computational efficiency of the proposed model.
The authors of this paper would like to thank the
AFRICA-ai JAPAN for funding this project under the
AFRICA-ai-JAPAN project innovation r