Deep learning algorithms have shown their great potential in the hyperspectral image (HSI) classification task. However, training these models generally requires a large number of training samples, which are laborious and time-consuming to collect in practice. To reduce the annotation burden, we propose to conduct HSI classification with point-level supervision, where only one annotated pixel in each category would be used for training. To this end, a novel region-growing fully convolutional network (RGFCN) is proposed in this study. The key idea of RGFCN is to expand the annotated regions from the original one point in each category with the region growing technique. As the annotated region grows, the network could also get stronger supervision, which in turn, helps to produce more accurate pseudo labels in the region growing. To better train the proposed RGFCN, we further adopt the entropy minimization strategy to assist the training in those unlabeled regions. Experiments on two benchmark HSI datasets demonstrate the efectiveness of the proposed approach.
As an important data source for Earth Observation,
hyperspectral images (HSIs) can record detailed
information from the Earth’s surface in both spatial and spectral
domains with hundreds or even thousands of continual
spectral bands. Because of this property, HSIs have been (a) (b) (c)
widely used in many applications, such as environment
monitoring, urban planning, and resource exploration Figure 1: An illustration of diferent types of annotations for
[
Witnessing the great development of deep learning annotation used in previous study.
algorithms in the computer vision field, recent research
has attempted to use advanced deep neural networks
to tackle the HSI classification task [
The main burden of assigning pixel-wise labels for a the class label. As shown in Figure 1 (b), the collection remote sensing image comes from the boundary regions of point-level annotation is much easier for an annotator of diferent objects. Figure 1 presents an example. It expert. can be observed that annotating the detailed boundaries Compared to the traditional classification scenario where the machine learning models could obtain sufifcient training data, classification with point-level supervision is much more challenging especially for supervised deep learning models which naturally require a large training set. Besides, the complex spatial distribution and the spectral heterogeneity of objects in HSIs
Probability Map Back-propagation Cross-Entropy Loss
Entropy Minimization Loss
…
Growing 8-Connectivity Neighborhoods
make it more dificult to achieve satisfactory classifica- neously. As the annotated training samples expand, the tion performance with weak supervision. To tackle the network could get stronger supervision, which in turn, aforementioned challenges, we propose a region-growing would also help to generate more accurate pseudo labels fully convolutional network (RGFCN) for HSI classifica- in the region growing. tion with point-level supervision. The initial inspiration Based on the aforementioned idea, we propose a of this work comes from an observation that adjacent region-growing fully convolutional network (RGFCN), pixels in remote sensing images, in particular those that as shown in Figure 2. In what follows, we will describe are of high spatial resolution, tend to belong to the same the proposed region-growing mechanism and the opticategory considering the spatial continuity of objects. mization of the network in detail. To this end, we propose a novel region-growing mechanism that helps to expand the annotated regions from 2.1. Region-Growing Mechanism the original one point in each category. As the annotated region grows, the network could also get stronger Formally, let ∈ Rℎ× × be the input HSI, where ℎ, supervision, which in turn, helps to produce more accu- , and are the height, width, and the number of bands rate pseudo labels in the region growing. To better train in the image, respectively. Let (· ) denote the mapping the proposed RGFCN, we further adopt the entropy mini- function of the backbone network, and = () ∈ mization strategy to assist the training in those unlabeled Rℎ× × be the corresponding probability map of , regions. Experiments on two benchmark HSI datasets where is the number of total categories. Recall that demonstrate the efectiveness of the proposed approach. our goal is to expand the annotated regions iteratively.
The rest of this paper is organized as follows. Section To this end, at each iteration in the training phase, we 2 describes the proposed RGFCN in detail. Section 3 visit all labeled pixels ∈ , where denotes the set presents the experiments in this study. Conclusions and that collects the locations of labeled pixels at the current other discussions would be made in Section 4. iteration. Specifically, for each ∈ , let ∈ [1, ] be its label and 8 denote the 8-connectivity neighborhood regions of . Then, for each unlabeled pixel (,) ∈ 8, the following criterion is used to update its label:
{︃argmax (︀ (,,′)︀) =
ℎ (,) ←
,
One of the main challenges of HSI classification with (,,) ≥ ,
point-level supervision comes from the insuficiency of (1)
training samples [
Alfalfa Corn-notill Corn-mintill Corn Grass–pasture Grass-trees
Oats Soybean-notill Soybean-mintill Soybean-clean Wheat
Woods Grass–pasture-mowed Hay-windrowed
Buildings-Grass-Trees-Drives Stone–Steel-Towers (a) (b) (c) (d) (e) (f) ∑︁ (,,) log (︀ (,,)︀) , randomly select 1 labeled pixel in each category from the (3) ground truth data to make up the original training set, while the remaining samples are used as the test set. where denotes the set that collects the locations of current unlabeled pixels. With the constraint in (3), the 1http://www.ehu.eus/ccwintco/index.php?title=
Hyperspectral_Remote_Sensing_Scenes ℒ on the labeled set ∑︁ (,,) log (︀ (,,)︀) , (2) ifrst define the cross-entropy loss as: ℒ = − 1
∑︁ || (,)∈ =1
where is the label map at current iteration.
Note that the annotations in would be very sparse
in practice since there are a lot of unlabeled samples. To
better train the network and apply constraint on those
unlabeled regions, the entropy minimization strategy is
adopted [
If the network has very low confidence in recognizing the class label in position (, ), its probabilities (,,) would be evenly spread in diferent classes. In this case, the entropy at position (, ) would also be large. Thus, to encourage the network make more confident predictions on those unlabeled regions, the entropy minimization loss ℒ can be defined as: ℒ = −
1 | | log ()
(,)∈ =1 predictions. lated as:
The final optimization of the network can be formu
min ℒ + ℒ, (4) where denotes the parameters in the backbone network, and is a weighting factor for the entropy minimization loss.
Note that the optimization of the network and the region growing procedure are simultaneous. At each iteration, we first use the back-propagation algorithm to update the parameters . Then, the labeled regions and the unlabeled regions get updated through the region growing mechanism.
3.1. Data Description The Pavia University dataset1, and the Indian Pines dataset1 are utilized to evaluate the performance of the proposed method. For each category in both datasets, we Model Backbone Only
RGFCN Backbone Only
RGFCN
In this subsection, we report the classification results of
the proposed RGFCN. Two diferent backbone networks
including the SpeFCN [
As shown in Table 1 and Table 2, the existing state-ofthe-art deep neural networks like SpeFCN and SpaFCN can hardly yield good performance in both datasets since there is only one labeled pixel for each category in the training set. By contrast, with the help of the proposed RGFCN, the performance gets dramatically improved. Take the result of the SpeFCN on the Indian Pines dataset for example. While the backbone network can only yield an OA of about 27%, the OA of RGFCN can reach more than 41%, which outperforms the previous one with about 14%. Similar phenomenons can be observed in other scenarios.
To visually evaluate the classification results, we further present the classification maps in Figure 3 and Figure 4. It can be observed that there exist a lot of salt and pepper noises in the results of SpeFCN and SpaFCN due to the insuficiency of training samples. By contrast, RGFCN can significantly improve the quality of the classification maps especially for objects with large spatial sizes like meadows. This paper proposes a region-growing fully convolutional network (RGFCN) for HSI classification with point-level supervision. To tackle the challenge of insuficient training samples, a novel region growing mechanism is proposed. Besides, the entropy minimization loss is adopted to further constrain the training on those unlabeled regions. Experiments on two benchmark HSI datasets demonstrate the efectiveness of the proposed method.
Since the performance of the whole framework depends largely on the quality of the region growing mechanism, once wrong annotations are included in the expanded labeled regions, the network may be misguided to make wrong predictions. Thus, we will try to further improve the region growing strategy to better filter out those unreliable samples in our future work.