The field of Earth observation is dealing with increasingly large, multimodal data sets. An important processing step consists of providing these data sets with labels. However, standard label propagation algorithms cannot be applied to multimodal remote sensing data for two reasons. First, multimodal data is heterogeneous while classic label propagation algorithms assume a homogeneous network. Second, real-world data can show both homophily ('birds of a feather flock together') and heterophily ('opposites attract') during propagation, while standard algorithms only consider homophily. Both shortcomings are addressed in this work and the result is a graph-based label propagation algorithm for multimodal data that includes homophily and/or heterophily. Furthermore, the method is also able to transfer information between uni- and multimodal data. Experiments on the remote sensing data set of Houston, which contains a LiDAR and a hyperspectral image, show that our approach ties state-of-the-art methods for classification with an OA of 91.4%, while being more flexible and not constrained to a specific data set or a specific combination of modalities.
Nowadays, our planet is monitored by a wide variety of
sensors that provide diferent information about Earth’s
surface [
An important processing step consists of providing these multimodal data sets with labels [5]. Manual laheterophily during propagation has only been solved theobjects that are at the same height, but when adding hy- label tend to be linked together or have a higher edge sive and time-consuming [6]. This brings up the need for belling demands for expert knowledge and is very expen- life, heterophily is rarely included in LP algorithms [11]. The classic LP methods additionally assume that all nodes Graph-based methods are preferred, since they can easlabel propagation (LP) methods for multimodal data [5]. in the graph are of the same type, thus creating a homooretically in [11] [13] [14]. None of these studies has goal is to assign the unlabeled nodes to one of the classes. performed any extensive tests on large, heterogeneous real-world data sets. This paper is organized as follows. Section 2 provides a
In Earth observation it can be challenging to find im- brief overview of the related works. Section 3 introduces ages that overlap both in space and time in order to con- the proposed method and Section 4 reports experimental struct multimodal data sets. In addition, certain imaging results. Section 5 discusses those results and Section 6 modalities are not available on large scale, due to limita- compares to state-of-the-art methods. Conclusions are tions of the imaging sensors [15]. This often results in drawn in Section 7. multimodal data sets that overlap only partially, which Throughout this paper, vectors (x) and matrices (X) raises the need for methods that are able to transfer dis- are denoted by lowercase and uppercase bold letters, recriminative information across modalities in order to spectively. refers to the element in the i-th column improve the classification of regions on Earth imaged by and j-th row of matrix X. a single sensor [15].
In this work we aim to develop an LP algorithm for multimodal remote sensing data. We focus on two questions: 1) How can the algorithm handle unimodal and multimodal data simultaneously and consequently also perform cross-modal information transfer? 2) How can homophily and heterophily interactions be estimated and included during LP?
Illustrative example Figure 1 sketches a simple example of the LP problem on multimodal remote sensing data with two diferent modalities: hyperspectral (HS) and LiDAR. The HS and LiDAR images are partly overlapping, which results in a heterogeneous graph with two node types: type 1 for the pixels with only HS data (’HS’), type 2 for the pixels in the overlapping region, with both HS and LiDAR data. In Figure 1 each node type is represented by a diferent shape, while colours are used to indicate the diferent classes a node can belong to.
Before LP, part of the nodes are unlabeled (white). The
In recent years, multiple attempts have been made to generalize standard LP algorithms to heterogeneous networks and/or to the heterophily property. Most research has been focussing on allowing heterophily during propagation. LinBP [14], CAMLP [16], and Relational Networks (RN) [17] include heterophily during LP, but cannot handle heterogeneous graphs. Heterophily is included using a propagation matrix H that defines how strongly diferent classes influence each other during propagation. The higher a value H(i,j), the larger the chance that a node of class i will be linked to a node of class j. A diagonal H reflects homophily, while a non-diagonal H reflects heterophily. Only three algorithms [13], [18], and [11] have attempted to generalize LP to heterogeneous graphs while simultaneously including propagation via heterophily. All three methods use multiple propagation matrices H to handle heterogeneous networks. ZooBP [11] appears to be the more general than [13] and [18], since it works on a fully heterogeneous graph (multiple node and edge types) and allows diferent node types to have a diferent amount of classes. The main drawback of ZooBP is that the propagation matrices H are not estimated during optimization but are required as input.
The following section describes our method for LP on multimodal remote sensing data. We first describe how the graph is constructed from the multimodal images, taking into account both the spectral and spatial information. Next, the propagation rules of ZooBP [11] are introduced, since they are used as a backbone in the new method. Last, we show how our method automatically estimates the propagation matrix H from any sparselylabeled graph, be it homogeneous or heterogeneous. This is a major improvement compared to previous graphbased LP methods such as ZooBP that require H as input.
each of a diferent imaging modality.
X has dimension dimension of modality X and Y, respectively. × and Y has dimension × , where and are the number of pixels and and are the spectral
When representing data as a graph, it is crucial to select an appropriate similarity measure that can grasp the underlying data structure [5]. Remote sensing data, and especially hyperspectral images (HSI), contain nonlinearities [19]. Furthermore, multi- or hyperspectral remote sensing data contain useful complementary information in both the spatial and spectral domain. Shi and Malik [20] introduce a graph construction technique that handles nonlinearity and simultaneously takes into account the spectral-spatial similarity between pixels (nodes). For each pair of pixels i and j, the similarity is calculated as follows: = −‖()−()‖ 22 ∗ { 0 −‖()−()‖ 22 if ‖() − ()‖ otherwise 2 < (1) the The left-hand side is the spectral proximity term, where () is the feature vector of the i-th pixel and width of the spectral kernel. The right-hand side is the spatial proximity, with () the spatial coordinates of the i-th pixel and
the width of the spatial kernel. Note that two nodes and will only be connected if they are less than r pixels apart. The spatial term prevents the formation of large disjoint communities in the graph, regardless of any spectral similarity [19]. We define two cases when building the graph.
Case 1 - Homogeneous Graph In this case we assume that the images X and Y fully overlap. For each pixel the diferent modalities are stacked together ( resulting in a homogeneous graph with N nodes. = + ),
Case 2 - Heterogeneous Graph In reality, however, multimodal remote sensing images often do not cover exactly the same region on Earth. When the images X and Y are only partly overlapping, some pixels carry information from both modalities, while other pixels carry information from only one modality. The result is a heterogeneous graph with a diferent node type for each modality or for combinations thereof.
The new LP method builds on the propagation rules of ZooBP [11], since they are flexible in terms of heterogeneity and heterophily. ZooBP [11] defines the following update equation to calculate all nodes’ beliefs in a single step: b ← e + (P − Q)b (2)
with e the prior belief vector and b the final belief vector, X and Y, obtained as follows: e = [vec (E1) … vec (E ) ]
and b = [vec (B1) …
vec (B ) ] . denotes the types. matrix for type-s nodes. set of node types occuring in the graph and a single node type. B and E are the prior and final (resp.) belief
P is constructed as follows: P = { }(,)∈{1,…,} 2; P ′ = ∑∈ ′ (H ⊗ W ) with ′ the set of edge types between nodes of type s and s’, the number of classes a node of type s can belong to, H the propagation matrix for edge type t, W the weighted similarity matrix for edge type t, and ⊗ stands for the Kronecker product. The interaction strength parameter allows to give more weight to particular edge The echo-cancellation matrix Q is defined as:
Q = ⨁=1 Q , where Q = ∑ ′∈ matrix of s-type nodes [11]. and ⨁ stands for the direct sum. D is the type-t degree ∑∈ ′ ′ 2 (H H ⊗ D )
H as an identity matrix, which reflects strong homophily [ 21]. The works that include heterophily either make use of domain experts to assign the values of H, as is the case for ZooBP [11], or use rarely-justified heuristics [ 21]. To overcome this issue, Kumar et al. [21] developed a method that estimates the propagation matrix H from a sparsely-labeled graph. However, it is only applicable to a homogeneous graph. In this section, we show how their method can be extended to the more general case of a heterogeneous graph.
Kumar et al. derive the energy function that belongs to the update equation of the label propagation method LinBP [14]. The derived energy function for LinBP is the following: ( B) = ‖B − E − WBH‖2, where B, E, W and H represent the same variables as described in Section 2 but used here for only a single node type. Since all variables are matrices, the Frobenius norm ||.|| is used. Next, the authors argue that for an unlabeled node i, the algorithm infers the final label by taking the average of its neighbours: Bi = (WBH)i, since there is no prior belief for this node. Based on the idea that the beliefs of a node are the average of its neighbours, the prior beliefs can be used to estimate H by minimizing the following energy function: ( H) = ‖E − WEH‖2.
Since ZooBP and LinBP use a similar update scheme, H for ZooBP can be estimated in an analogous way [14]. The main challenge is to adapt the energy function to a heterogeneous graph, where a diferent matrix
to be estimated for each edge type t. When dropping the echo-cancellation term Q, as is done in [21], the update equation of ZooBP for a node of type s becomes: B = E + ∑
∑ ′∈ ∈ ′ W B ′H Following the same strategy as in [21], while also taking into account that the ZooBP theory requires H to be residual, results in the following constrained optimization problem:
minimize ( Htot) = ‖Etot − WtotEtotHtot‖2 subject to ∶ ∑ Ht(, ) = ∑ Ht(, ) = 0 ∀, , (3) (4) Where Htot, Etot, and Wtot are global matrices defined as: Etot = (
E1, ..., Es), Htot = (Ht)∈ ′ = (Hij)1≤,≤ and Wtot = (Wt)∈ ′ = (Wij)1≤,≤ . The constraint enforces H to be residual, which means that all rows and columns sum to 0 [11].
In this section we assess the general performance of the
new LP algorithm. The classification is quantitatively
evaluated using the overall accuracy (OA, in %) and the
Kappa coeficient ( , in range [
, , and r. Optimal values are set via a random search using 3-fold cross-validation on the following sets: a real number in the range [0.001, {0.001, 0.01, 0.1, 1, 10} and r an integer in the range [0, in the set {0.001, 0.01, 0.1, 1, 10},
in the set
The method is evaluated on the publicly available multimodal remote sensing data set of Houston [23] to verify the efectiveness of the proposed approach. The data set contains a HSI and a LiDAR-derived DSM image of an urban area in Houston, the USA. The images have a spatial size of 349 x 1905 pixels with a resolution of 2.5 m.
The HSI consists of 144 spectral bands, ranging from 380 to 1050 nm. The data set contains 15 classes of interest and is visualised in Figure 2. The training and test set contain 2832 and 12197 samples, respectively.
The optimal values for the hyperparameters are found to be: = 0.317, = 1,
= 10 and = 18. The proposed method produces a smooth classification map to their ground truth label. Due to computational limi- labeled regions, more specifically where spatially adjatations, during the experiments the test set is split into 6 folds in a stratified fashion. Each fold is classified using a separate graph and classification of the full test cent pixels belong to diferent classes. E.g. the classes ’running track’ and ’synthetic grass’ or ’stressed grass’ and ’healthy grass’ often get confused. These are pairs set is obtained by combining the results of the individ- of classes with a similar spectral signature. The combi
The results demonstrate that the proposed method achieves good classification performance. Misclassification mainly happens for spatially near pixels that have a similar spectral signature. In that case the hyperspectral data is not able to diferentiate between two classes and that is when the LiDAR data should help to make the right decision. Table 1 However, since the two modalities are fused on data-level, the LiDAR data is nearly insignificant in the feature vector that is mainly composed of hyperspectral features. An adaptive feature selection method should be considered that decides for each pixel or data set which modalities should weigh more than others in the decision-making contributes to: OA (%) 78.6 Classification accuracies on Houston obtained by state-of-theart approaches. Reported: overall accuracy (OA, in %) and Kappa coeficient (no unit). process.
The large spatial size of remote sensing images causes scalability issues in our method, since the similarity matrix W is computed by comparing each pair of data points. Note, however, that this scalability issue is only related to size of the input images, and not to any intrinsic methodological problem. The number of iterations until reaching convergence is constant for increasing graph sizes, which proves that the propagation rules scale well for large graphs.
The methods listed in Related Works are not able to solve the general problem (full heterogeneity and heterophily) and are therefore left out of comparison. Instead, we focus on the application side and provide a comparison of the proposed method with five state-of-the-art classiifcation methods: a support vector machine (SVM) with radial basis function (RBF) kernel [24], a Random Forest (RF) [25], ’3-stream CNN’ [26], ’patch-to-patch CNN’ (PtoP CNN) [27], and ’sparse and low-rank component analysis’ (SLRCA) [4]. A seed size of 30% is used here in our method to provide fair comparison. Also note that 3-stream CNN, PToP CNN and SLRCA classify the whole data set at once, which is advantageous for the classification performance. Our method largely outperforms the SVM and RF and (practically) ties the other three classification methods. Moreover, the proposed method can easily accommodate other types of multimodal data, as well as supporting a large variety of complex graph structures, which are two major advantages compared to [26], [27], and [4] which are tailored to specific remote sensing modalities and/or data sets.
We have designed and evaluated a new label propagation method for multimodal remote sensing data that • Generality: The method works on a fully heterogeneous graph (multiple node and edge types) and includes homophily and/or heterophily interactions via the propagation matrices H. • Estimation of propagation strengths: The propagation matrices H are automatically estimated from any homogeneous or heterogeneous sparsely-labeled graph. No need anymore for domain experts to assign these values. • Cross-Modal Information Transfer: The proposed method can propagate information between unimodal and multimodal data points.
The method achieves similar performance as state-of-theart approaches, while being more general. Future work therefore includes testing the algorithm on multimodal data sets originating from other domains to check the robustness of the new approach. Furthermore, attention will be paid to solving the scalability issue that arises when using graphs with thousands of nodes.
This work is funded in part by Centre for Integrated Remote Sensing and Forecasting for Arctic Operations (CIRFA) and the Research Council of Norway (RCN Grant no. 237906), and the Automatic Multisensor remote sensing for Sea Ice Characterization (AMUSIC) Framsenteret ”Polhavet” flagship project 2020. manova, N. N. Alekseeva, M. A. Arshinova, E. Y.
to Assess the Degree of Transformation of the Land Cover Structure, IOP Conference Series: convolutional neural networks, Remote Sensing 10 (2018) 1649. doi:1 0 . 3 3 9 0 / r s 1 0 1 0 1 6 4 9 . [27] M. Zhang, W. Li, Q. Du, L. Gao, B. Zhang, Feature Extraction for Classification of Hyperspectral and LiDAR Data Using Patch-to-Patch CNN, IEEE Transactions on Cybernetics 50 (2020) 100–111. doi:1 0 . 1 1 0 9 / T C Y B . 2 0 1 8 . 2 8 6 4 6 7 0 .