Having the ability to quantify prediction confidence or uncertainty will greatly assist the successful integration of deep learning methods into high-stake decision making processes. Graph-based convolutional neural networks can be trained to perform classification of multimodal remote sensing data using a model output which represents a Dirichlet distribution parameterization. This parameterization can then also be used to obtain measures of prediction uncertainty. By making a correspondence between a multinomial opinion, as described by subjective logic, and a Dirichlet distribution parameterization, a direct mapping between the two can be performed. A multinomial opinion of this kind can produce quantified measures of uncertainty and distinguish uncertainty due to a lack of evidence (vacuity) and uncertainty due to conflicting evidence (dissonance). With an appropriately chosen loss function, the graph-based classifier will converge to provide accurate estimates of uncertainty. The results presented in this paper show that the measures of uncertainty provided by such models are capable of better distinguishing out-of-distribution data samples than probabilistic measures of uncertainty produced by equivalent deterministic neural networks.
ter results than any individual data mode would produce in isolation. Each data capturing technique will natuThe capability of algorithms to provide accurate measures rally have its own strengths and weaknesses, inherent to of confidence and uncertainty is important if they are to the physical properties of the sensing mode [6, 7]. Debe adopted in real-world scenarios where the stakes can terministic classification, while useful, is held back by be high [1]. Although deep learning methods are often its inability to express uncertainty. Adoption of such capable of producing high-accuracy predictions [2, 3], techniques will always be limited by the adopter’s trust they are generally criticized for being unable to express in the predictions. Uncertainty estimates, however, will when to have confidence in the prediction and when greatly assist human trust in models, as it will provide the prediction should be presented as uncertain. If deep a quantification of confidence that might indicate when learning models are to be integrated reliably into real- a prediction is not to be trusted, and more importantly, world decision making processes, it is of vital importance when a prediction is given with great certainty [8]. that the methods being used are capable of accurately In this paper, we have analyzed how well diferent meaexpressing uncertainty [4]. sures of model uncertainty perform the task of identify
With remotely-sensed data being available with ever- ing data points which belong to a distribution other than greater temporal and spatial resolutions, the development those observed during training (out of distribution detecof computational processing methods which are capable tion). To do so, we have used graph-based neural network of robustly handling such large volumes of data will assist architectures that are adapted to provide subjective opincountless earth-monitoring applications [5]. Specifically, ions (as described in the field of belief or evidence theory with data being captured now using a wide range of [9]) through the use of Dirichlet distribution parameteritechniques with complementary strengths, the ability to zations [10, 11]. The subjective opinions can be used to combine this data into a multimodal analysis will allow measure two intuitive measures of uncertainty: vacuity each data mode to interact synergistically to provide bet- and dissonance. Vacuity is a measure of the uncertainty related to an absence of observed evidence, i.e. a higher measure of vacuity suggests a lack of supporting evidence for a prediction. Dissonance is a measure of prediction uncertainty arising due to the presence of conflicting evidence. This approach (using graph-based neural networks within a subjective-logic framework) is, to the best of our knowledge, as-yet untested as a method for performing classification of multimodal remote sensing data. The performance of the adopted technique represents a promising avenue in the search for meaningful uncertainty estimates for this task.
The remainder of this paper is organized as follows: Section 2 describes the uncertainty framework adopted in the methods presented, Section 3 details the construction of the graph-based neural networks used, Section 4 presents an analysis of results and Section 5 summarizes and draws conclusions as well as suggests areas for future work.
The proposed uncertainty-aware framework relies on the definition of uncertainty metrics, which in turn are based on subjective logic and a Dirichlet mapping [11]. These steps are detailed in this section, and have been properly adapted to the task of multimodal remote sensing classification.
Subjective Logic (SL), takes an evidence-based approach to decision making [12]. Expressing an opinion using measured quantities of belief allows the distinction to be made between uncertainty due to a lack of evidence (vacuity) and uncertainty due to the presence of conflicting evidence (dissonance). A multinomial opinion, , can be expressed as = (b, , a), where b is a belief mass vector, the scalar is the uncertainty mass and a is the base rate vector. For a -class classification problem, y, a and b are all vectors of dimension . A projection of onto a probability distribution can be made according to
( = ) = + .
It follows that since ∑︀
=1 = 1 for the base rate vector, an additivity requirement is described by + ∑︁ = 1.
=1
If p is a -dimensional random vector containing the probability of belonging to each output class, and is the strength vector which parameterizes a Dirichlet distribution, the probability density function of the Dirichlet is given by
Dir(p| ) = ∏︀=1
=1 ) ∏︁ Γ( ∑︀
− 1, =1 where Γ() is the gamma function. The distribution’s expected value is given by
E [Dir(| )] = ∑︀ =1 .
If we allow the uncertainty mass and base rates to be given by and =
∑︀ =1 = = 1/, ∀ () ≡ = , (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) respectively, where refers to the Dirichlet strength, then by equating the probability projection of (1) with the expected value of the Dirichlet distribution given by (4), the expression for the belief mass can be obtained as = − 1 .
This provides us with everything needed in order to map from a Dirichlet distribution to a SL opinion and vice versa.
From the definitions of the evidential uncertainties presented in [9], the measures of vacuity and dissonance have been adopted. The measure of vacuity uncertainty is simply given by the uncertainty mass, i.e. and the measure of dissonance uncertainty is given by (︃ ∑︀̸= Bal( , ) )︃ () = ∑=︁1 ∑︀̸= where Bal() is a function which gives the relative balance between two belief masses, defined by
Bal( , ) = {︃1 − 0, |− | , if + ̸= 0, + otherwise.
The entropy of the node-level multinomial distributions provided by the models is also computed to represent a form of uncertainty. This is done in order to provide a comparitive metric against which the evidential uncertainties can be compared.
The multimodal data can be represented using a graph, where each of the nodes in the graph represents a pixel in the image. The graph’s adjacency matrix, A ∈ R× , by D = ∑︀
A .
A set of features, X ∈ R× , is used to assign a vector description of each graph node, where denotes the number of input features. The graph’s degree matrix, D ∈ R× , is a diagonal matrix with elements given
The graph convolutional networks (GCNs) used are of the form proposed by [10], where the graph convolutional layer is given by
Z(+1) = ︂( ˜ − 12 A˜ D˜− 21 Z() () D ︂) , (11) where Z(), Z(+1) and () are the inputs, outputs and weights of the th layer respectively, and () is a nonlinear activation function. For brevity, the tilde operator in the graph, i.e. A˜ = A + I and D˜ = D + I. is used to represent the inclusion of self-connection edges
An adaptation to the GCN architecture used by [10] must be made in order to obtain the subjective opinions that will be used to obtain measures of vacuity and dissonance uncertainty. The adaptation made means that the where E = − rameters, such that the output will provide a probability distribution over multinomial class probabilities for each node. To do so, the softmax output activation function used in the output layer of the GCN is substituted for a ReLU function. In this way, the model is trained to output non-negative evidence contributions, E ∈ R× , 1 and refers to the -dimensional concentration parameters of the th node. In order to train such a model, the loss function is made up of two components: a squared error term, which is minimized in order to classify a greater proportion of the nodes correctly, and a variance term, which is minimized to incentivize the model to provide confident predictions where possible. This loss, ℒ( ), is given by ℒ( ) = ∑︁ ∑︁ [︀ ( − )2 + Var()︀] , ∈L ( − )2 + 2 ︂( −
︂)] − , (12) = ∑︁ ∑︁ ∈L
︂[ where ∈ L refers to the fact that the loss is computed using a sum only over nodes in the training set, L. Models trained with such an output activation and loss function will be denoted using the ‘S-’ prefix in order to indicate they provide subjective predictions, e.g. S-GCN.
In order to assist the convergence of subjective models, two additional assistance techniques have been used: is used to represent edges between nodes deemed similar. teacher knowledge distillation and the use of a Dirichlet model will output node-level Dirichlet distribution pa- convergence of a subjective graph convolutional model. prior. These have been shown to allow subjective models to provide better uncertainty estimates [11].
By training a non-subjective model in advance, its outputs, ˆ, can be used in order to encourage the subjective model to converge to node Dirichlet distributions with E[] which are close to the teacher’s deterministic estimates. This is achieved using an additional term in the loss function, ℒT( ) = ∑︁ ∑︁ ︂( ˆ log
ˆ E[] ︂) , (13) which corresponds to the summation of Kullback-Leibler (KL) divergence terms between the teacher output probability and the expected value of the subjective model’s Dirichlet distribution for each node. Using KL( || ) to compute the KL divergence, this is stated equivalently as ∑︀ KL(ˆ ‖ E[]). Notice that this sum is computed over all nodes as opposed to just the nodes in L. Models trained using a teacher are denoted using the ‘-T’ sufix e.g. a S-BGCN-T model would indicate that a pre-trained GCN was used as a teacher in order to assist the training
A second convergence assistance technique which can be used involves the use of a Dirichlet prior, ˆ . The exact method chosen to provide ˆ will depend on the nature of the problem but we will assume nodes which are nearby in the graph are more likely to belong to the same output class than nodes which are far apart, a property known as homophily [13]. Using this assumption, we can use the computed distances on the graph to assign contributions of evidence from observed node labels to the other nodes in the graph using a function of our choosing. If denotes the shortest path distance between a given node, indexed by and an observed node, indexed by , then the amount of evidence contributed to suggest that the th node belongs to the th class is given by ⎨ ⎧ ⎪⎪ exp ︃( − 2 )︃
2 2 ⎪⎪⎩0, otherwise, ℎ( , ) = (2 2)1/2 , if = 1, (14) where is a scale parameter which controls the order of distance magnitude over which evidence will propagate in the prior. The total evidence to suggest the th node belongs to the th class, can be found by summing these contributions over the nodes in the training set, such that the element in the prior is given by ˆ = 1 + = 1 + ∑︁ ℎ( , ).
(15) ∈L which can, in turn, be incorporated into the total loss function. Models trained using a prior are denoted using the ‘-K’ sufix.
Table 1 shows how these convergence assistance techniques can be weighted and combined in various permutations to provide a total loss function, ℒtotal( ), as well as the model name abbreviations used to denote which combination has been used. The ‘B’ in the model names of Table 1 refers to the fact that dropout inference has been used as a Bayesian approximation. The coeficients T and K are used to control the relative importance of the teacher network and the Dirichlet prior respectively against the importance of the subjective loss function given in (12). These have been considered as hyperparameters which are to be tuned during training. -nearest neighbors algorithm with two nodes receiving an edge connecting them if either node was one of the nodes which were nearest the other. This produces a graph which is both undirected and unweighted. The graph, which contains approximately 2.16 million nodes, was computed with = 15.
In order to measure an uncertainty output’s ability to separate OOD nodes, a receiver operating characteristic (ROC) curve and a precision-recall (PR) curve can be computed. The area under the ROC curve and PR curve (AUROC and AUPR respectively) can be used as a single numerical representation of the detection performance, where an area of 1.0 would represent a perfect discriminator for both metrics.
hyperparameters
Models were implemented and trained using the TensorFlow library [15] on a personal laptop computer with 4.1. Data Intel Core i7 CPU and 16 GB of RAM. In order to handle the imbalance of classes in the dataset, sample weightA subsection of the 2018 IEEE GRSS Data Fusion Chal- ing was used. Samples were given weights which were lenge dataset [14] ws selected for the purposes of validat- inversely proportional to the number of total samples ing the described methods. The ground truth labels in of each class in the training set. This allows the losses this dataset describe 20 diferent urban land cover/land related to nodes from under-represented classes to have use classes (i.e. = 20) as well as an unlabelled state, an increased influence over parameter updates and vice described as Unclassified. The modes of input data repre- versa. sent measurements from three sensor types: LiDAR, opti- All GCN-based models were constructed using a cal and hyperspectral (HS). The LiDAR data was provided dropout layer (dropout probability 0.5), a graph conat 0.5 m resolution, the same resolution as the ground volutional layer, as described in (11), a second dropout truth labels (GT). In order to simplify analysis, the optical layer (dropout probability 0.5) and a second graph condata (which was provided at 0.05 m resolution) and the volutional layer with the relevant output activation funcHS data (which was provided at 1.0 m resolution) were tion. The kernel weights of the first graph convolutional bilinearly resampled to obtain 0.5 m resolution across layer were regularized using an 2 penalization. Where inputs and outputs. dropout inference has been used, the number of samples
The graph was constructed with each 0.5 m × 0.5 m taken was 100. pixel representing a node in the graph. Each node has Hyperparameters including the learning rate, the a 52-dimensional feature vector describing it (produced 2 regularization coeficient and the number of GCN by stacking 3 optical channels, 48 HS channels and 1 layer output features, , were selected via a grid-search LiDAR channel). The graph edges are computed using a
It would be reasonable to expect that uncertainty should be higher when the model is asked to make a prediction using an input which does not resemble the inputs upon which it was trained. The relative inability of neural networks to successfully extrapolate beyond the support of the training data is a well-known weakness of these methods [16]. By training models using only a subset of the classes provided by the GT, with the other classes acting as out of distribution (OOD) samples, the OOD detection ability of the uncertainty metrics can be measured. The AUROC and AUPR can be calculated for each uncertainty output provided by each model type, in order to determine the relative performance of the respective metrics for this task.
In the results presented, two classes were randomly selected to act as OOD. This was repeated 10 times, with two new randomly sampled classes selected for each training and evaluation loop in order that the variation in OOD detection performance due to the nature of the classes selected as OOD could be averaged out and the mean and standard deviation computed. Each model type was assessed over the same 10 sampled OOD class pairs for fairness. The AUROC and AUPR values measured can be found in Table 2.
In this paper we have adapted a novel classification method capable of providing uncertainty estimates to the task of multi-class classification of multimodal remote sensing data. The adopted framework, based upon the theory of Subjective Logic, provides measures of vacuity and dissonance uncertainty. Of the types of uncertainty assessed, the measure of vacuity was the best metric to perform identification of OOD samples. Experimental results have shown the performance of the S-BGCN-TK model in the task of OOD detection to be improved against baseline methods. This represents a promising avenue for uncertainty-aware learning in the task of multimodal remote sensing classification.
The presented results illustrate the importance of convergence assistance techniques as a means for improving the quality of uncertainty estimates, particularly through the use of a prior. This can be seen by comparing the SBGCN-T-K OOD detection performance with equivalent models which do not use a prior, e.g. S-BGCN-T.
Future work should consider the generalisation potential of this method by assessing performance on other World Congr. DAIS - Work. Distrib. Anal. Infraschallenging remote sensing classification datasets. The truct. Algorithms Multi-Organization Fed., 2017, pp. analysis could also be extended to assess whether the 1–6. doi:10.1109/UIC-ATC.2017.8397411. presented uncertainty measures could be used to detect [9] A. Josang, J.-H. Cho, F. Chen, Uncertainty charactermodel misclassifications. Additionally, there is scope for istics of subjective opinions, in: 2018 21st Int. Conf. research into how the choice of method for computing Inf. Fusion, Cambridge, U.K., 2018, pp. 1998–2005. the ˆ prior afects the quality of uncertainty estimates, [10] T. N. Kipf, M. Welling, Semi-supervised classificaeither by varying the scale parameter, , or considering tion with graph convolutional networks, in: 5th Int. diferent prior computation methods entirely. Conf. Learn. Representations, Toulon, France, 2017. [11] X. Zhao, F. Chen, S. Hu, J.-H. Cho, Uncertainty aware semi-supervised learning on graph data, in: Acknowledgments Advances Neural Inf. Process. Syst., volume 33, 2020, pp. 12827–12836.
This work is funded in part by Centre for Integrated [12] A. Jøsang, Subjective Logic - A Formalism for ReaRemote Sensing and Forecasting for Arctic Operations soning Under Uncertainty, Artificial Intelligence: (CIRFA) and the Research Council of Norway (RCN Grant Foundations, Theory, and Algorithms, Springer, no. 237906), the Automatic Multisensor remote sensing for Sea Ice Characterization (AMUSIC) Framsenteret ‘Pol- [13] 2Q0.1H6.udaonig:1, 0H..1H0e0,7A/.9S7i8n-g3h-,3S.1-9N-. 4L2im33,7A-. 1B.enson, havet’ flagship project 2020, the Isaac Newton Trust, and Combining label propagation and simple models Newnham College, Cambridge, UK. out-performs graph neural networks, in: 9th Int. Conf. Learn. Representations, 2021.
References [14] S. Prasad, B. Le Saux, N. Yokoya, R. Hansch, 2018 IEEE GRSS Data Fusion Challenge - Fusion of Mul[1] B. Goodman, S. Flaxman, European Union reg- tispectral LiDAR and Hyperspectral Data, 2018. ulations on algorithmic decision-making and a URL: https://dx.doi.org/10.21227/jnh9-nz89. doi:10. ‘right to explanation’, AI Mag. 38 (2017) 50–57. 21227/jnh9-nz89.
doi:10.1609/aimag.v38i3.2741. [15] M. Abadi, et al., TensorFlow: Large-scale machine [2] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, learning on heterogeneous systems, 2015. URL:
Nature 521 (2015) 436–444. https://www.tensorflow.org/, Software available [3] L. Zhang, L. Zhang, B. Du, Deep learning for remote from tensorflow.org.
sensing data: A technical tutorial on the state of [16] B. Lakshminarayanan, A. Pritzel, C. Blundell, Simthe art, IEEE Geosci. Remote Sens. Mag. 4 (2016) ple and scalable predictive uncertainty estimation 22–40. using deep ensembles, in: Advances in Neural Inf. [4] J. D. Lee, K. A. See, Trust in automation: Design- Process. Syst., volume 30, Long Beach, CA, USA, ing for appropriate reliance, Human Factors 46 2017, pp. 6402–6413. (2004) 50–80. URL: https://doi.org/10.1518/hfes.46.1.
50_30392. doi:10.1518/hfes.46.1.50_30392. [5] M. Chi, A. Plaza, J. A. Benediktsson, Z. Sun, J. Shen,
Y. Zhu, Big data for remote sensing: Challenges and opportunities, Proceedings of the IEEE 104 (2016) 2207–2219. [6] S. Chlaily, M. D. Mura, J. Chanussot, C. Jutten,
P. Gamba, A. Marinoni, Capacity and limits of multimodal remote sensing: Theoretical aspects and automatic information theory-based image selection, IEEE Trans. Geosci. Remote Sens. 59 (2021) 5598–5618. doi:10.1109/TGRS.2020.3014138. [7] A. Marinoni, S. Chlaily, E. Khachatrian, T. Eltoft,
S. Selvakumaran, M. Girolami, C. Jutten, Enhancing ensemble learning and transfer learning in multimodal data analysis by adaptive dimensionality reduction, CoRR abs/2105.03682 (2021). URL: https: //arxiv.org/abs/2105.03682. arXiv:2105.03682. [8] S. Chakraborty, et al., Interpretability of deep learning models: A survey of results, in: IEEE Smart