Remotely sensed imagery is a well-established data source for spatial predictions, however,it comes with some disadvantages. Here, we explore how another data source could be used for spatial predictions, namely, geospatial vector data, by looking at a specific case study: LULC classification. We show how vector data can be encoded for an artificial neural network, making it geographicly aware and enable it to predict LULC classes. We use two diferent encoding schemes as well as two diferent artificial neural network architectures. Our results suggest that geospatial vector data can be used for LULC classification and that the type of encoding and artificial neural network plays a significant role.
a region and contain descriptions such as cofee shop , or, A series of spatial prediction tasks, such as land use supermarket. To be able to create a deep learning model, land cover (LULC) classifications, are based on remotely the artificial neural network (ANN) architecture needs sensed imagery as input data. Electromagnetic re- to be able to deal with this irregular geographic and selfectance of the physical surface enables to gain knowl- mantic data. Here, we use LULC classification as a case edge of the Earth and to base prediction models based study to explore how geospatial vector data can be used on it. However, using remotely sensed imagery comes in combination with ANNs for spatial prediction tasks. with two major disadvantages: (1) Electromagnetic re- To be able to deal with the irregular nature of geospatial lfectance does not capture all relevant geographic phe- vector data, we employ two encoding techniques which nomena, such as land use, and may even be inconsistent transform the vector input data to a feature space that across multiple sensor types and scenes [1]. (2) The raster can be used by an ANN. The outcome of this work is model introduces two assumptions into the model: That two-fold: (1) We show that geospatial vector data can be the world can be tessellated in a regular manner and is ho- used for LULC classification; (2) we show that the type mogeneous within a pixel [2]. As such, the raster model of encoding used for transforming geospatial vector data also does not maintain topological characteristics of real- as well as the ANN architecture have an impact on the world objects [3]. In this work we probe the usage of ifnal classification performance. The first section briefly another data source for this purpose, namely geospatial discusses the problems which appear when using geospavector data, which is expressed in geometric primitives tial vector data as input for ANNs. Then, we describe (polygon, line strings, points) and attributes (e.g. type of the experiments carried out and provide the preliminary object, height, etc.) describing corresponding geographic results. In the end, we discuss our findings and outline objects (geo-objects). future research.
Geospatial vector data has attributes that make it a challenging task to be used as input for a deep neural network. A deep neural network requires structured and well-defined data, whereas vector data is highly irregular.
Unlike raster data, (geospatial) vector data is irregular. A raster dataset tessellates a region of interest (ROI) into pixel, which is an ideal data model to be used for various deep learning architectures, as it is already formatted in a desired regular format that can be forwarded to an ANN right away. In contrast, geo-objects, such as a house, a street, a garden, a river, etc. are described as geometric 3. Methodology primitives. A polygon for example might describe the shape of a house, a linestring a street, or, a point geometry the location of a point of interest. Next to these We used two input datasets for the classification experigeometric descriptions, geo-objects contain attributes, ments over Austria, which is the ROI: (1) Geospatial vecwhich describe the corresponding geo-object. Such at- tor data from LinkedGeoData; (2) CORINE LULC raster tributes can be, for example, the type of geo-object, such data. There are 13 diferent LULC classes in Austria, deas a cofee shop , a bar, or a university. This data model fined by CORINE Level 2, described by grid-cells of size may allow for a precise description of geographic and 100m x 100m. For each LULC class, 12,000 grid-cells were extensive auxiliary information (attributes) but is irregu- selected randomly, each of which serves as ground truth lar and cannot be used in its native form by ANNs. This for the corresponding sample. Then, for each of the gridirregularity has two aspects: (1) a geometric and (2) a cells, geospatial data from LinkedGeoData was obtained semantic aspect. for the area 1km around the corresponding cell center.
Ad 1: Training samples would have a varying num- This data would then ultimately be used to predict the ber of geo-objects. ANNs however require input vectors LULC class of the corresponding LULC grid-cell. Howwith a fixed number of input features. Furthermore, geo- ever, in order two be able to do so, the vector data was objects come in three diferent geometric primitives – transformed. Two encoding schemes were used for this ANNs can also not deal with this aspect, as they do not purpose, the Geospatial Configuration Matrix [ 6], and have the capabilities to represent this geometric informa- an encoding that was developed within the realm of this tion without loss. work. We denote this encoding as the Channel Encoding.
Ad 2: Semantics of a geo-object, which describe it, are often of nominal scaling (e.g. restaurant, or university). 3.1. The Geospatial Configuration Matrix However, an ANN requires numerical input, not a nominal one. One may simply transform each nominal value The Geospatial Configuration Matrix (GSCM) computes to a token number. However, this would introduce a flaw local statistics of every present class of geo-objects (see into the ANN-based model as the distance between any ifgure 1b for an illustration). It represents the mean, two token values will introduce a wrong semantic sim- minimum, maximum of the distances and azimuths of all ilarity in feature space. For example, cofee shop might geo-objects of a particular type, relative to the LULC gridhave a value of 128, restaurant a value of 1076 and school cell center. For example, all geo-objects of type restaurant a token value of 203. These values would suggest that a around a LULC grid-cell (distance is 1km) are obtained, cofee shop is semantically more similar in feature space then, all azimuths and distances between the restaurants to school than to restaurant. Such a tokenization of nom- and the LULC grid-cell are computed. Afterward, the inal values is a standard procedure in natural language mean, minimum and the maximum of these distances processing (NLP) [4] and can be used to train language re- and azimuths were computed, which provides 6 features lated models by embedding words. However, geographic with regards to class restaurant. There are 805 geo-object data, which is used here, does not exhibit a sequential classes within the ROI, which were not defined by us, character such as a sentence observed in NLP. There- but by the ontology of LinkedGeoData. As a result, the fore methods from NLP are not suitable to model the GSCM provides 4830 (6 features x 805 potential classes) geographic distribution of semantic features. The encod- features for each sample. LinkedGeoData describes the ings used here overcome this issue and cherish the two geo-objects with an Web Ontology Language (OWL) ondimensional (geographic) distribution of the semantics. tology. As such, each geo-object can be an instance of
In this work, we used two encoding schemes which multiple classes. For example, a pet-shop is also a shop as enable us to transform geospatial vector data into a fea- well as an amenity. ture space that tackles these two problems. As such, we take geospatial vector data from LinkedGeoData1, which 3.2. The Channel Encoding describes each geo-object semantically using an ontology, and aim at predicting LULC over Austria. Prior work suggests that geospatial semantics are highly associated with LULC classes [5]. The ground truth is obtained from CORINE (Level 2)2.
coding, was developed within this work (see figure 1a).
The focus of this encoding is to represent the geographic distribution of the geo-objects while avoiding the problem of a wrong semantic similarity. For this purpose, an image stack having a width and height of 1km, 805 channels and a pixel resolution of 10m x 10m was created for each sample. Each channel represents the geographic distribution of one OWL class of geo-objects, as each dmax a2 d3cejd1 a3 d2 (b) The GSCM computes six local statistics for each class of geo-objects around a grid-cell (which is obtained from CORINE level 2). The distance threshold around a grid-cell is denoted and has to be defined. Here, we use a threshold of 1km only. In this figure, the diferent shapes denote diferent classes of geo-objects. For example, all pentagons could be of class Restaurant. Azimuths (1 to 3) and distances (1 to 3) are used to calculate six descriptive statistical values for this class of geo-objects. Source of image: [6] we calculated the overall accuracy as well as kappa coefifcient for each of these four experiments (two encodings times two networks). Please note that the hyperparameters for each encoding were the same. In order to find a set of suitable hyper parameter, a random search was performed, as literature suggest its superiority over a systematic search, such as a grid search [9]. The chosen hyper parameters can be seen in table 1. (a) The Channel Encoding scheme. Each layer in the image stack represents the geographic distribution of geo-objects of a particular OWL class (e.g. pet-shop). Each pixel represents the number of geographic objects of this OWL class that appear within the corresponding pixel. Here, dark blue pixels denote higher numbers whereas light blue pixels denote a lower number of geo-objects. pixel value is the number of geo-objects of this particular class which are present within the same pixel. 3.3. The artificial neural network
architectures Within this work we employ two diferent ANN architectures: The MLP-Mixer [7] and the Perceiver architecture [8]. The MLP Mixer architecture is based on fully connected layers only, however, they are applied layer-wise 4. Results, Analysis, and and pixel-wise. As such, it is a competitive alternative to convolutional neural networks and visual transformer Discussion networks, which are usually utilized for image vision tasks. The Perceiver architecture has two main char- Table 2 shows the results of the accuracy assessment of acteristics: (1) It uses Transformer mechanisms which, the four experiments. Using the Channel Encoding and in addition, use a reduced latent space as a bottleneck. the Perceiver yielded the highest accuracy of 72.0% with a This enables to speed up the computation. (2) It uses a kappa coeficient of 0.69. However, using the Channel Enparametrized Fourier feature encodings which enables coding yielded the lowest accuracy once it was used with to retrieve attentions, based on relative positions. As the MLP-mixer network. In contrast, using the GSCM such, the Perceiver architecture is designed as a general- yielded the highest accuracies once used with the MLPpurpose ANN and was utilized for image data, sound Mixer (overall accuracy of 70.2% and kappa coeficient of data, and 3D point clouds [8]. 0.68). The Perceiver architecture and Channel Encoding may have yielded the best results as the Perceiver is a 3.4. The experiments general-purpose ANN which can handle a high number of channels. Additionally, the Perceiver uses a Fourier Once each encoding was performed for the input data, feature encoding, which is parametrized. As such, it can they were forwarded to the two ANNs used in this work, compare values based on relative positions. This capanamely, the MLP-Mixer and Perceiver network. Finally, bility enables the network to learn meaningful spatial relationships which are present within the entire extent
of the Channel Encoding. In contrast, the MLP-Mixer does this hyper parameter to future research. not apply such strategies which can potentially explain the poor performance when the Channel Encoding is used with it and might suggest that attention mechanisms combined with learnable position encoding can significantly help to detect meaningful spatial relationships. Another characteristic of the Perceiver might have improved the accuracies, once the Channel Encoding was used: it is permutation invariant to its input. As such, certain spatial constellations may have appeared in a transformed manner (e.g. rotated or shifted) but still be captured as related and meaningful. A potential explanation for the MLP-Mixer to have a lower classification performance when using the Channel Encoding than when using the GSCM, could be the following: The MLP-Mixer model transforms every patch of the input into a latent space, using a multilayer perceptron (MLP). This patch size is set by the number of channels (805 in the case of the Channel Encoding) and a defined width and height. Here, we used the entire extent of 2km (two times 1km) as width and height, covering the entire extend of a sample. While it might be more dificult for the MLP to compute a meaningful latent representation for a bigger patch than for a smaller one, a smaller patch size might hamper to find meaningful spatial relationships. Thus, the MLP-Mixer model might not have been able to compute meaningful latent representations as the input patches were too References big. In this work we explored only one dmax threshold (1km). However, [6] found that this value can have a significant impact on the performance of the spatial prediction model. As such, we would expect that the LULC classification performance would change here too, once dmax is changed. However, we leave the exploration of
tor data can be used for a spatial prediction task like LULC classification. They also indicate that the way how geospatial vector data is encoded plays a crucial role. Also, the type of network plays an important role: The Perceiver network was able to learn meaningful features from the Channel Encoding, which was not the case with the MLP-Mixer. In this work, we described a first step towards how geospatial vector data can be encoded for using it with deep neural networks. As such, future research can explore novel ways how vector data can be eficiently encoded. Furthermore, ANN architectures that do not require a prior encoding could be explored. In this work we explored using vector data alone, however, future research could focus on how vector data and imagery, such as hyperspectral [10] or multispectral optical imagery [11], could be fused into one enhanced feature space. [2] P. Fisher, The pixel: A snare and a delusion, International Journal of Remote Sensing 18 (1997) 679–685. doi:10.1080/014311697219015. [3] S. Winter, Unified Behavior of Vector and Raster Representation, University of Technology Vienna, 2000. URL: https://books.google.at/books?id= QfGwcQAACAAJ. [4] J. J. Webster, C. Kit, Tokenization as the initial phase in nlp, in: COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics, 1992. [5] M. M. Cutchan, I. Giannopoulos, Geospatial Semantics for Spatial Prediction, in: S. Winter, A. Grifin, M. Sester (Eds.), 10th International Conference on Geographic Information Science (GIScience 2018), volume 114 of Leibniz International Proceedings in Informatics (LIPIcs), Schloss Dagstuhl–LeibnizZentrum fuer Informatik, Dagstuhl, Germany, 2018, pp. 45:1–45:6. doi:10.4230/LIPIcs.GISCIENCE. 2018.45. [6] M. Mc Cutchan, S. Özdal Oktay, I. Giannopoulos, Semantic-based urban growth prediction, Transactions in GIS 24 (2020) 1482–1503. URL: https://onlinelibrary.wiley. com/doi/abs/10.1111/tgis.12655. doi:https: //doi.org/10.1111/tgis.12655. [7] I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, A. Dosovitskiy, Mlp-mixer: An all-mlp architecture for vision (2021). URL: https://arxiv.org/abs/2105.01601. arXiv:2105.01601. [8] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, J. Carreira, Perceiver: General perception with iterative attention (2021). URL: https: //arxiv.org/abs/2103.03206. arXiv:2103.03206. [9] J. Bergstra, Y. Bengio, Random search for hyperparameter optimization, J. Mach. Learn. Res. 13 (2012) 281–305. [10] Y. Chen, H. Jiang, C. Li, X. Jia, P. Ghamisi, Deep feature extraction and classification of hyperspectral images based on convolutional neural networks, IEEE Transactions on Geoscience and Remote Sensing 54 (2016) 6232–6251. doi:10.1109/ TGRS.2016.2584107. [11] Y. Chen, C. Li, P. Ghamisi, X. Jia, Y. Gu, Deep fusion of remote sensing data for accurate classification, IEEE Geoscience and Remote Sensing Letters 14 (2017) 1253–1257. doi:10.1109/LGRS. 2017.2704625.