The use of deep learning for water extraction requires precise pixel-level labels. However, it is very dificult to label highresolution remote sensing images at the pixel level. Therefore, we study how to utilize point labels to extract water bodies and propose a novel method called the neighbor feature aggregation network (NFANet). Compared with pixel-level labels, point labels are much easier to obtain, but they will lose a lot of information. In this paper, we take advantage of the similarity between the adjacent pixels of a local water body, and propose a neighbor sampler to resample remote sensing images. Then, the sampled images are sent to the network for feature aggregation. Our method uses neighboring features instead of global or local features to learn more representative features. The experimental results show that the proposed NFANet method not only outperforms other weakly supervised approaches, but also obtains similar results as the state-of-the-art ones.
ture extraction is highly dependent on the availability of
suficient pixel-level labels for training. However,
highWater-body extraction from high-resolution remote sens- resolution remote sensing images are large in scale and
ing images is an important research topic in the field data volume, which makes pixel-level labeling extremely
of remote sensing. Although the traditional algorithms laborious. The pixel-level annotation usually requires a
have made some progress in water-body extraction, there lot of time and labor costs, as well as professional
knowlare still problems such as low automation, cumbersome edge to accurately mark uncertain boundaries between
manual feature extraction, and insuficient extraction diferent classes of interest, which hinders the extraction
accuracy. In recent years, deep learning has become of informative features from high-resolution remote
sensan emerging research hot spot in the field of artificial ing images to a certain extent. Training models using
intelligence. The rapid development of deep learning weak labels have received more and more attention in the
technology and the improvement of computer hardware ifeld of computer vision. Compared with fully-supervised
performance have made deep learning, especially the semantic segmentation, weak-supervised learning does
CNN-based techniques, successful in many important not require pixel-level labels, and has the characteristics
tasks, such as image classification, target detection, and of fast labeling and low cost. However, the use of weak
semantic segmentation, and their performance has sur- annotations makes the supervision information seriously
passed many traditional algorithms. The work [
Unlike other natural objects, water bodies are usually the local information or global information of an image, liquid, the colors and textures of local water bodies are the neighbor feature aggregation efectively utilizes the very similar. Therefore, there is a high degree of simi- neighbor information and, therefore, more representative larity between neighbor pixels in water bodies, which features can be learned. makes the inherent diference of the features contained between the neighbor pixels of the water-body generally smaller than that of the non-water-body. We hope to 2. Method map the neighbor pixels of the remote sensing image to the same location in space, and then extract neighbor Figure 1 illustrates the proposed weakly supervised wafeatures from multiple neighbor pixels, and use the neigh- ter extraction framework. Figure 1.a shows the entire bor features to jointly decide whether the pixel at this recursive training process. We will describe it in the location belongs to the water bodies. Based on the above third section. The acquisition of pseudo-labels is shown motivation, we propose the neighbor feature aggrega- in Figure 1.b. We input neighbor images into the network tion network (NFANet) to make full use of this property. and use point labels for supervision to obtain neighbor Specifically, we utilize a sampling method called neigh- features. Then the feature aggregation module is used to bor sampler to generate a set of neighbor images from aggregate the features extracted from the previous step. high-resolution remote sensing images. The neighbor Finally, post-processing is performed to obtain pseudopixels of the original image are separately allocated to labels. We will describe the details of each of the above each neighbor image, so that the pixel values of any two steps in the following sections. neighbor images at the same position are similar but the pixel values are diferent. On the whole, neighbor image 2.1. Neighbor Sampler groups have similar but diferent characteristics. Then, we use an end-to-end model to perform feature extraction on each image of the neighbor image groups, and aggregate the features by using the feature aggregation module. Compared with other methods that only use
bor images group (1 () , 2 () , . . . , ()) from a single optical remote sensing image . represents the number of neighbor images. Figure 2 shows the schematic diagram of generating a group of neighbor
channels of each neighbor feature to one. CMax pooling is defined mathematically in detail as follows: Given a three-dimensional feature maps tensor group = (1 () , 2 () , . . . , ()) ∈ R× × × , The operation of CMax pooling is as follows: ,,() = (,,1,(), ,,2,(), . . . , ,,3,()), (1)
= 1, 2, . . . , , = 1, 2, . . . , , = 1, 2, . . . , .
As a result, the feature maps group = (1 () , 2 () , . . . , ()) ∈ R× × is obtained. Then, the OTSU algorithm is used to binarize each feature in to obtain the result = (1 () , 2 () , . . . , ()) ∈ R× × . The formula is as follows: images using the neighbor sampler. Let us assume that the width, height, and channel of the input image are bo,rsa, mp, lreerspect=ive(ly.1T,h2e,i.m. p.l,eme)nitsatdioenscorfibtehde anseifgohl-- = () , = 1, 2, . . . , . (2) lows: Finally, we vote for all binarized neighbor features of the 1. The image is divided into × cells, where neighbor feature group to obtain the aggregated result the size of each cell is × × . is experimentally . is calculated using the following equation: set to 2 and, therefore, = × = 4.
2. For the − ℎ row and − ℎ column of the cell, the {︃1, ∑︀ 2 pixels in the adjacent positions of each cell are selected , = 0, ∑︀=1 ,, ≥ (3) in the order from top to bottom and from left to right, =1 ,, < 2 which are regarded as the (, ) − ℎ elements of = To sum up, the mathematical definition of the feature (1 () , 2 () , . . . , ()). When is set to 2, the aggregation module is detailed as follows: pixels at the upper left, upper right, lower left, and lower right adjacent positions are selected, respectively. = ( ( ( ))) (4) 3. For all × cells being divided in step 1, step 2 will be repeated until all the cells are resampled, and where ∈ R× × × represents the neighbor feaa neighbor sampler = (1, 2, . . . , ) is generated. tures group and ∈ R× is the output. Next, the Given an optical remote sensing image , neighbor im- aggregated result is input to the post-processing modages group (1 () , 2 () , . . . , ()) is generated, ule. The specific operations include filling small holes in where the size of each neighbor image is × × . the closed area by using area filling and removing noise
In this way, the neighbor image dataset can be gen- by using morphological operations. Then, we apply a erated from the original dataset. Neighbor images are point-label constraint to the processed results. If the area similar but not identical, because for any two neighbor in the result contains point labels, the entire area is reimages, (, ) − ℎ pixel comes from the neighboring tained, otherwise it is not retained. The generated results location of the original remote sensing image. are used as pseudo-labels and input into the recursive training as supervision information.
post-processing
We input the neighbor images group to an Recursive training is a weakly supervised strategy. When
end-to-end network to extract features, and ob- applying the resulting model over the training set, the
nettain the corresponding neighbor feature group work outputs capture the shape of objects significantly
(1 () , 2 () , . . . , ()) ∈ R× × × , where better than that of just pseudo-labels [
applied the method to high-resolution visible spectrum
images for water extraction. This water-body dataset
comes from the Gaofen Challenge [
In the experiment, the weak supervision models use point labels as the initial supervision information, while other weak supervision methods can predict the local the full supervision models use pixel-level labels. Because area of the water body, there are errors in the detection of the remote sensing image segmentation/classification the water body boundary, while our method is relatively evaluation index of overall accuracy or Kappa coeficient more accurate. The studied weak supervised methods cannot efectively describe the real structure of image cannot detect small objects appropriately while this issue segmentation geometry, we choose to use fgIoU (fore- is solved to a great extent by the proposed method. ground IoU), bgIoU (background IoU), mIoU (mean IoU), fgDice (foreground Dice), bgDice (background Dice) and 3.4. Efectiveness of neighbor sampling mDice (mean Dice) to comprehensively evaluate the results. For each model, we performed five independent In the ablation experiment, the other settings are unruns to calculate the aforementioned evaluation indica- changed, and only the value of is changed. We set the tors and standard deviations. neighbor sampling parameter k of our proposed network from 1 to 4 and only use cross entropy and dice loss to 3.2. Comparison with Fully Supervised train the model. For diferent values of NFANet, in order to avoid interference from other modules, we only
Approaches select UNet as the feature extraction network for comIn Table 1, we report the water extraction performance parative experiments. In particular, when the value of of our proposed approach and compare it with the fully is set to 1, the neighbor image group degenerates into supervised approaches. Figure 3 also provides the visual the input image. As shown in Figure 5, with the gradual performance of all approaches. These approaches ran- increase of , mIoU first increases and then decreases. domly use 70% of the samples as the training set, and With the increase of neighborhood sampling paramethe remaining data as the test set. Experiments demon- ter , the number of adjacent pixels to be considered strate that our method achieves the best score using the increase geometrically, resulting in information redunNestedUNet-based model, and the visual performance dancy. The size of each reconstructed neighbor image is shows that the prediction results obtained by our method gradually reduced, and the boundary of the water body are very close to the ground truth. The mIoU of our will also become unclear. Therefore, we set equal to 2, method reached 75.22%, and the mDice reached 85.04%. because the neighbor features require less computation Compared with the best fully-supervised model DeepLab and achieves better performance.
V3+, the mIoU of our method is only reduced by 3.03%, and mDice is only reduced by 2.23%. But the labeling 3.5. Efectiveness of feature aggregation cost of our method is much less than that of the fully supervised method. Nevertheless, it is dificult to achieve fully supervised performance using only point labels.
pervised remote sensing approaches. The experimental results are shown in Table 2. To be fair, all methods are based on UNet. It can be seen that the mIoU of our method is 8.84%, which is higher than that of the U-CAM-based method with the mDice of 6.89%. In addition, Figure 4 shows the prediction results of other weakly supervised methods and our method. Although As shown in Table 3, when is set to 2 ( = 4), assuming that the features of the − ℎ neighboring image is , ∑︀
=1 means feature aggregation is used. Compared with the best method that does not use feature aggregation, the mIoU and mDice of our method are improved by 4.8% and 3.7% respectively. To a certain extent, the greater the number of neighboring images, the more features are available, and these features can complement each other. Therefore, after the feature aggregation, the performance of the prediction results can be improved.
paper consisted of Intel Core i7-9700k 3.60 GHz CPU, GeForce RTX 2080Ti GPU, and 16GB RAM. The results of the GPU inference time are shown in Table 4. The results in the table are the average GPU inference time of the data set. After using recursive training to improve the quality of pseudo-labels, we input the pseudo-labels and original images into the models consistent with the fullysupervised methods for training and inference. Therefore, the GPU inference time of the proposed method is the same as that of the fully-supervised methods. It can be observed from the table that the inference time of DLinkNet is the shortest. because DLinkNet compresses the feature channel in the decoder to reduce the computational cost. NestedUNet embeds U-Nets of diferent depths in its architecture, which requires more convolution calculations, thus increasing the consumption of inference time.
Unlike traditional convolutional neural networks that only use global or local features for discrimination, NFANet uses neighbor features, which allows more representative features to be learned. We fuse these neighbor features to obtain pseudo-labels, and improve the label quality by recursive training. We tested it on water data sets and compared it with advanced fully supervised and weakly supervised methods. By using only point labels, the proposed method obtains comparable results with that of full supervision. As a possible future work, we will conduct research on weakly supervised or semisupervised methods of self-correction. Remote sensing images collected from satellites are usually afected by spectral variability.
Work in [