Satellite imagery is a form of data that can be used for many applications, especially those focusing on change over time. In this article, we analyze methods of detecting buildings and predicting their height as well as what key attributes are required for good predictions. Building detection and prediction are done by using neural network algorithms such as convolutional neural networks to estimate their height. Predictions are made based on additional data of the building and its area. In this paper three diferent building estimation models are implemented. The research showed that using a mixed dataset that takes both Sentinel image patch data and numerical feature input of additional building data performs well even with lower quality images.
focus is ice, and all-year measurements of clouds and aerosol distributions over land in Polar Regions [7].
Having a way of quickly predicting building height pro- Sentinel-1/Sentinel-2 images came from the Copernivides us with essential knowledge for sustainable urban cus programme which is a European initiative for the development and plays a vital role in the fields of urban, implementation of information services dealing with the pollution transmission, building energy consumption, environment and security, based on observation data repopulation estimation [1]. Essentially building height ceived from Earth Observation (EO) satellites and groundinformation is crucial for the comprehensive understand- based information. Copernicus API provides access to ing of urban development [2]. Satellite images obtained during the Sentinel missions
Determining urban development and its magnitude allowing comparison of the same locations within difusually requires the aggregation of many criteria. This is ferent time frames. The Sentinel-2 satellite is equipped a dificult process due to the time it takes to access this with an opto-electronic multispectral sensor for surveyinformation and the possible changes that might happen ing with a resolution of 10 to 60 m in the visible, near while the data is being collected. Even with automated infrared (VNIR), and short-wave infrared (SWIR) spectral monitoring systems, this creates linked data which is zones, including 13 spectral channels. This ensures the hard to handle [3]. In addition, some things that are not capture of diferences in vegetation state, including temdocumented or finished will not be collected and this poral changes, and minimizes impact on the quality of will make the resulting predictions less accurate, as with atmospheric photography. The orbit is an average height the degradation of the data accuracy, predictive accuracy of 785 km and the presence of two satellites in the misgoes down too [4]. Therefore, approaching this problem sion allows repeated surveys every 5 days at the equator there is a need to use data that is easier to get and reflect and every 2-3 days at middle latitudes. the current state precisely. Satellite images that are up An analysis of literature where Satellite images are to date are a great source of current data that is freely used has shown a variety of diferent use cases, such available. as detecting specific crops and change in the soil struc
There are many sources of satellite images like Lidar, ture [8]. Other examples include continuous observation Sentinel-1, Sentinel-2, InSar, ICESat and others. All of of ships moving on the sea surface [9] and retrieving these specialize in diferent areas, have diferent spec- significant wave height [10]. trums that can be used for diferent tasks. For example, Li- The focus of this work is on prediction of building dar and InSar capture ground elevation/deformation and height. There are already existing approaches that help are great for tasks that focus on nature/surface changes to solve similar problems. For example, shadows in com[5][6]. ICESat images measure ice sheet balance. The bination with gradient formulas are employed in highIVUS 2022: 27th International Conference on Information Technology resolution images for building height extraction [11], *Corresponding author. objects that are salient are identified and then their edges © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License are found [12]. Multi-scene building height estimation CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) method is using shadow length calculation combined getting the building imagery data “Copernicus Open Acwith fish net and Pauta criterion [ 13]. Other building cess Hub” applied programming interface (API) was used detection works include using the U-Net model to assign [22]. This allows getting satellite image patches with adsemantic labels to each pixel as building/non-building ditional information (latitude, longitude, time of the year [14] or using 3D building models in conjunction with and the day that the pictures were taken). To further simsatellite imagery to predict building height [15], as well plify the process “Copernicus Open Access Hub” applied as using deep convolutional neural networks (DCNN) for programming interface (API) was used with “Python” semantic segmentation and applying filters [16 ]. programming language integration. This has facilitated
Convolution neural networks (CNNs) networks could the download and processing of Sentinel-2 images. An be one of the most promising options for building de- example of taken image can be seen below (see Figure 1). tection and height prediction. CNN models have been proven to be good at extracting mid and high-level abstract feature representations from small raw images [17] for classification purposes, by interleaving convolutional and pooling layers, i.e., spatially shrinking the feature maps layer by layer. Recently proposed network architectures also allow for dense per-pixel predictions[18].
Many of these models rely on high-resolution satellite imagery, the detail of which makes it easier for the models to identify and detect extremely small elements [19][20].
The spatial resolution of high-resolution satellite images is about 1m/pixel [20]. The downside, however, is that these high-resolution images are not taken very often.
This creates the preconditions for working with old data, which could potentially give a false picture of the current Figure 1: Sentinel-2 example satellite image of Barcelona situation. Therefore, lower quality satellite imagery is a more appropriate choice in this case.
In this work Sentinel satellite imagery with medium- Areas in Europe were selected by bounding mentioned resolution 10 m/pixel Sentinel images [21] has been used geographical zone in a two-point rectangle (58°59’42.0"N which allows to see the development of housing and 10°14’20.4"W x 36°57’00.0"N 36°25’51.6"E). An example infrastructure. The authors of this study performed ex- can be seen below (see Figure 2). According to bounds periments combining Sentinel imagery with additional “Planet OSM” data query was adapted to only select buildgeographic and time data in order to achieve feasible ings in the bounded area. Also, buildings that were less building height estimation accuracy. than 20m in height were filtered out.
Finally work is composed as follows: the details described in Chapter II include the process of collecting images of the buildings, their height and additional data, as well as the peculiarities of forming a complete data set.
Chapter III describes three diferent models that were used to estimate the height of the buildings. Chapter IV provides a comparative analysis of the results of these models. Chapter V provides summaries and insghts from the study.
bound to Europe. This gives the ability to have diferent types of buildings without covering the whole world.
For the administrative information (building location, Figure 2: Bound geographical zone with “geojson.io” height, area) "Planet OSM” database was used which contains complete copies of the “OpenStreetMap” database. “OpenStreetMap” (OSM) is a collaborative project to create a free editable geographic database of the world. For
The “geojson.io” and building polygon (coming from “Planet OSM” database) coordinate data was used to download large Sentinel images of the region. Polygon data from “OpenStreetMap” was iterated and patches of 32x32px were cut out by centering patch to the center of building polygon. This helped to build image dataset consisting of 32x32px image patches with building in the center. Also, metadata was taken from the image (latitude, longitude, time of the year and the day the pictures were taken). Then image patch data was concatenated with administrative building data. Each image of a building is accompanied by its height and area in square meters (for non-linear model). The dataset creation flowchart can be seen below (see Figure 3).
more evenly distributed dataset, heights that have lower amount of images were removed. Building height 0.5 quantile was calculated to be 30m and therefore buildings of height above 30m were removed from the dataset (see histogram in Figure 4). model with an error of 1.89 m. The nonlinear regression model was in the middle among Sentinel-based models with an error of 2.02 m (see Table 2). Here, the mean absolute error (MAE) refers to how many meters each forecast difered from the actual height of the building. detection. Different models schemes provided in above According to the results of models predictions on test (see Figure 6). set of buildings in range 20-30m it is visible that without
Neural network based models architectures can be having any additional information (only that comes with seen in below (see Figure 7). On the left side (Model Sentinel imagery - latitude, longitude, time of the year 2) - the CNN model that takes an image of size 32x32 (0-365) and the time of day (0-24) that the pictures were pixels and returns a height prediction. On the right side taken) it is possible to predict building height with 1.89m (Model 3) - the Mixed data model is shown, similarly takes MAE. This is on average 13cm more precise than using a 32x32 pixels image but additionally takes additional additional data of building area which can be not availnumerical data latitude, longitude, day of the year and able if building is newly built or not yet registered. Also, time of the day (input3 of size 4) and concatenates image the mixed data model is on average 42cm more precise and numerical data layers to single layer that returns than CNN image model. predicted height. Both models try to minimise the mean squared error loss function (MSE). arenas, the nonlinear regression model provides a more aerial optical and aerial light detection and ranging (Liaccurate prediction than other models based on image dar) data to prepare the training data. Both models use a recognition. In the photo shown, the shadow angles the convolutional neural network (CNN) architecture. The building, making it larger. It is possible that for this IM2ELEVATION model takes a single optical image as inreason, the CNN model predicts a higher height for the put and produces an estimated DSM image as output. The building. As seen in the prediction results of building 2 IM2ELEVATION model achieved a mean absolute error example (see Figure 9), the mixed data model is the most of 1.46 while the mixed data prediction model achieved a accurate. This could be that additional data of latitude mean absolute error of 1.89. The Mixed data model, howand longitude helps the model to consider what other ever, makes use of satellite photos of lesser quality than buildings are in the surrounding area and thus making the areal ones. The Mixed data model outperforms the the prediction more accurate. By the results on building 3 IM2ELEVATION model where buildings are sparsely dis(see Figure 9) the most accurate prediction is also made tributed in the scene. As it performs better when building by mixed data model. are not as close together and have more distinct features.
line model. The Mixed data model consisting of CNN
DenseNet201 backbone feature extractor was trained on
This could be due to the fact that there is only one a dataset that takes both Sentinel image patch data and clearly separated building from other buildings in the numerical feature input of additional building data. It area with visible shadow and there are some surrounding performed better than both the baseline model and CNN buildings for mixed data reference. DenseNet201 backbone model that used satellite imagery
To furthermore check the correctness of the results alone. more diverse dataset should be made. This dataset should Comparing the performance of the trained models, it contain images of larger range of heights not only 20- turned out that the mixed data prediction model provides 30m with similar distribution of images between heights. the best results. A mean absolute error of 1.89 and a mean Also it would be wise to test how models perform on absolute percentage error of 0.079 were achieved. The areas that have dense/sparse building distribution. Fea- reason for better performance could be that additional ture importance tool checker, such as [24], can be used data of latitude and longitude helps the model to consider to determine whether part of a CNN model identifies a what other buildings are in the surrounding area and thus building shadow as an important feature/property. make the prediction more accurate.
For additional information we compared the Mixed A limitation of the models is that they are trained data model to the IM2ELEVATION model used for mainly only on data from the southern Europe area and Building Height Estimation from Single-View Aerial feature buildings only up to 30 meters. When estimatImagery [25]. Both models use diferent d atasets, the ing buildings from diferent regions, this could result IM2ELEVATION model uses a multisensory fusion of in inconsistent results. The area was chosen due to the abundance of data, while the height limit was decided ary Satellite. Sensors 2021, 21, 7547. https://doi.org/ to ensure a more evenly distributed dataset, as smaller 10.3390/s21227547. building data greatly outnumbers large building data. [10] Xue, Sihan, et al. "Significant wave height retrieval
Testing the model with low buildings showed that to from Sentinel-1 SAR imagery by convolutional neuapply the same model to smaller buildings a more high- ral network." Journal of Oceanography 76.6 (2020): quality dataset is needed. To improve predictions, images 465-477. of buildings that are not as close together are required, [11] Raju, P. L. N., Himani Chaudhary, and A. K. Jha. as they have more distinct features. "SHADOW ANALYSIS TECHNIQUE FOR EXTRACTION OF BUILDING HEIGHT USING HIGH RESOLUTION SATELLITE SINGLE IMAGE AND ACReferences CURACY ASSESSMENT." International Archives of the Photogrammetry, Remote Sensing and Spatial [1] Lívia Tomás, Leila Fonseca, Cláudia Almeida, Fer- Information Sciences (2014).
nando Leonardi, Madalena Pereira (2016) Urban [12] X. Cai, H. Sui, R.Lv, and Z. Song, “Automatic circular population estimation based on residential build- oil tank detection in high-resolution optical image ings volume using IKONOS-2 images and lidar data, based on visual saliency and Hough transform” In: International Journal of Remote Sensing, 37:sup1, Proc. of IEEE Workshop on Electronics, Computer 1-28, DOI: 10.1080/01431161.2015.1121301. and Applications, pp.408-411, 2014. [2] Mahtta, Richa, Anjali Mahendra, and Karen C. Seto. [13] Xie, Yakun, et al. "Multi-Scene Building Height Es"Building up or spreading out? Typologies of urban timation Method Based on Shadow in High Resolugrowth across 478 cities of 1 million+." Environmen- tion Imagery." Remote Sensing 13.15 (2021): 2862. tal Research Letters 14.12 (2019): 124077. [14] Irwansyah, Edy, Heryadi, Yaya, Agung, Alexan[3] Lim, Chiehyeon, Kwang-Jae Kim, and Paul P. der. (2021). Semantic Image Segmentation Maglio. "Smart cities with big data: Reference mod- for Building Detection in Urban Area with els, challenges, and considerations." Cities 82 (2018): Aerial Photograph Image using U-Net Models. 86-99. 10.1109/AGERS51788.2020.9452773. [4] Bansal, Arun, Robert J. Kaufman, and Rob R. Weitz. [15] David Frantz, Franz Schug, Akpona Okujeni, Clau"Comparing the modeling performance of regres- dio Navacchi, Wolfgang Wagner, Sebastian van der sion and neural networks as data quality varies: A Linden, Patrick Hostert. "National-scale mapping business value approach." Journal of Management of building height using Sentinel-1 and Sentinel-2 Information Systems 10.1 (1993): 11-32. time series." Remote Sensing of Environment 252 [5] Cheng Wang and Nancy F. Glenn Integrating Li- (2021): 112128.
DAR Intensity and Elevation Data for Terrain Char- [16] Niemeyer, Joachim, Franz Rottensteiner, and Uwe acterization in a Forested Area. IEEE Geoscience Soergel. "Contextual classification of lidar data and and Remote Sensing Letters · August 2009. building object detection in urban areas." ISPRS [6] Kang, Y.; Lu, Z.; Zhao, C.; Xu, Y.; Kim, J.-W.; Galle- journal of photogrammetry and remote sensing 87 gos, A.J InSAR monitoring of creeping landslides (2014): 152-165. in mountainous regions: A case study in Eldorado [17] Zhou Feiyan, Jin Linpeng and Dong Jun, "Review National Forest, California. Remote Sensing of En- of convolutional neural networks", Journal of comvironment 258 (2021): 112400.D. Harel, First-Order puter science, vol. 40, no. 6, pp. 1229-1251, 2017. Dynamic Logic, volume 68 of Lecture Notes in Com- [18] Bearman, Amy, et al. "What’s the point: Semantic puter Science, Springer-Verlag, New York, NY, 1979. segmentation with point supervision." European doi:10.1007/3-540-09237-4. conference on computer vision. Springer, Cham, [7] Zwally, H.J.; Schutz, B.; Abdalati, W.; Abshire, J.; 2016.
Bentley, C.; Brenner, A.; Bufton, J.; Dezio, J.; Han- [19] Tobler W. “Measuring Spatial Resolution” 1987. cock, D.; Harding, D.; et al. ICESat’s Laser Measure- https://www.researchgate.net/publication/291877360 ments of Polar Ice, Atmosphere, Ocean, and Land. Measuring spatial resolution.
J. Geodyn. 2002, 34, 405–445. [20] IKONOS-2. https://earth.esa.int/web/eoportal/satellite[8] Lang, Nico, Konrad Schindler, and Jan Dirk Wegner. missions/i/ikonos-2.
"Country-wide high-resolution vegetation height [21] Resolution and swath. mapping with Sentinel-2." Remote Sensing of Envi- https://sentinel.esa.int/web/sentinel/missions/sentinelronment 233 (2019): 111347. 2/instrument-payload/resolution-and-swath. [9] Yu, W.; You, H.; Lv, P.; Hu, Y.; Han, B. A Moving [22] Copernicus Open Access Hub.
Ship Detection and Tracking Method Based on Op- https://scihub.copernicus.eu/. tical Remote Sensing Images from the Geostation- [23] Botchkarev A., “Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology” 2018.
https://arxiv.org/abs/1809.03006. [24] Ribeiro M. T., Singh S., Guestrin C. “Why Should I
Trust You?” Explaining the Predictions of Any Classifier. 2016. https://arxiv.org/pdf/1602.04938v1.pdf. [25] Liu C-J, Krylov VA, Kane P, Kavanagh G, Dahyot R.
IM2ELEVATION: Building Height Estimation from Single-View Aerial Imagery. Remote Sensing. 2020; 12(17):2719. https://doi.org/10.3390/rs12172719