In this paper, two approaches to building extraction from satellite imagery and height data obtained from stereo images or LIDAR are compared. The first approach consists of detecting high-rise objects in a digital surface model and then improving recognition accuracy using segmentation of spectral information. The second approach uses the U-Net convolutional neural network, which showed the best results for the extraction of objects from aerospace images on a number of large datasets. Extensive experiments were carried out to evaluate the dependence of the quality of U-Net-based building extraction on the different data types (including high-resolution satellite images and digital surface model data). Building extraction quality of the trained network was also evaluated on satellite images with different spatial resolutions.
At the present time, monitoring the state of the Earth's surface based on aerospace images plays an important role in many fields such as agriculture, construction, emergency analysis and others. This approach has a number of undeniable advantages: timeliness, large coverage of areas and a relatively low cost. At the same time, many tasks are handled manually by an operator, which makes it important to develop tools for automatically retrieving required information from satellite images.
The extraction and classification of terrain objects are the most important tasks in processing satellite imagery. The solutions of such problems find their application in cartography (update of topographic maps), the detection of changes in the composition of the terrain objects (i.e. analysis of urban development and illegal deforestation), navigation of aircrafts and others. However, the methods existing at the moment extract and classify objects of a terrain from space images with insufficient accuracy. First of all, this is due to the complexity and variability of the scenes under consideration, which contain a huge number of objects of different nature significantly influenced by shooting conditions. All this contributes to the importance of research aimed at overcoming these difficulties and developing software systems for processing remote sensing data.
In this paper, in order to improve the quality of the building extraction, a digital surface model
(DSM) containing heights of a terrain and objects on it was applied. Height information for the DSM
can be obtained by shooting a terrain from two different angles. Two such images form a stereo pair,
which is passed to the input of a stereo matching algorithm, for calculating a matrix of pixel shifts (a
disparity map) belonging to the same objects between two images. After that, the digital surface
model of the terrain is constructed by calculating the heights of the terrain and objects on the basis of
the disparity map and the parameters of shooting. In this paper, we used a digital surface model
obtained as a result of the operation of the stereo matching algorithm proposed in [
The paper is organized as follows. The second section describes an approach to building extraction,
based on the detection of high-rise objects of a terrain from a DSM and the refinement of their
boundaries on the basis of segmentation of spectral information. The third section is devoted to the
U-Net [
As a starting point for building extraction, we consider a segmentation algorithm, which uses heights of objects in addition to the spectral information to provide a better segmentation of satellite images.
Segmentation is used to pre-process images in many computer vision tasks, since it allows analyzing
homogeneous (by certain criteria) regions instead of separate pixels of an image. The characteristics
used to split an image can be very different: the color, the texture, etc. To date, a large number of
segmentation algorithms have been developed. In [
To test the influence of height information on the result of satellite image segmentation, an
algorithm based on the idea of grouping pixels in regions on the basis of the similarity of their
brightnesses and heights was presented in [
1. Median filtering of the image.
2. Preliminary selection of a region. For each pixel in the image we form a region by recursive search for neighboring pixels of similar height and color. After that, we calculate mode values of the height and each of the four image channels in the obtained region and find in it a pixel having a height exactly equal to the calculated mode of height and the closest brightness to the modes of each channel.
3. Selection of the region. Next, from the found pixel, we form a new region by repeating recursive search, and fill the new region with the color of the computed mode values.
4. Repeat steps 2-3. Due to the noisiness of the input data, the regions obtained in the previous steps will have a small area. The repetition of the segmentation procedure will allow merging the regions into larger ones.
5. Small region removal. As a result of the brightness or height discontinuity, regions with a negligible area can remain. Such regions are combined with neighboring ones in such a way that the region to which the pixels are added has a larger area, and its height and brightness are as close to the added pixels as possible. This procedure is performed for regions in order of increasing area.
After segmenting the satellite image (using the algorithm mentioned above or another one), it is required to determine whether the obtained segments belong to buildings.
To do this, it is necessary to find objects in the DSM that rise above the terrain. Such objects include buildings and forest tracts. For this purpose, the following algorithm is proposed.
1. Search for height differences. For each pixel p in the DSM with the height value h, we consider its one-dimensional neighborhoods →( ) and ↓ ( ) of the radius in horizontal and vertical directions. For the pixels ∈ ( ), we calculate ∆ℎ = ℎ − ℎ +1, ∈ [− , − 1], where the index 0 corresponds to the pixel , which is the center of the neighborhood. And if ∆ℎ0 = max ∆ℎ or Image Processing and Earth Remote Sensing
∆ℎ0 = min ∆ℎ , and |∆ℎ0| ≥ , then the pixel is marked as having a significant height difference. The threshold specifies the minimum height of buildings to be detected. We do not consider the pixels at which the minimum and maximum are simultaneously reached, since they are the errors in the DSM.
2. Obtaining high regions. From the pixels that have a significant height difference, we construct line segments, which will form the target regions. For the construction of these line segments, we search horizontally and vertically in forward and backward directions in the DSM for pixels with a height difference upward. Then we begin to draw a line segment from each selected pixel that goes: to the pixel with the height difference upward, the value of which is added to the value of the difference at the starting pixel. Then the construction of the line segment continues; to the pixel at which the height value is less or close to the height value at the beginning of the line segment; until the line segment reaches the fixed length.
Performing this procedure in the forward and backward directions (from left to right and vice versa, from top to bottom and vice versa) allows us to find even those regions of buildings in which one of the sides can be blurred due to the influence of noise.
Next, the pixels of the segments selected only horizontally or vertically are discarded. The remaining pixels of the segments form regions. We select only those of them that contain at least one pixel with a significant height difference. This operation allows rejecting the false intersections of the vertical and horizontal segments.
3. Refinement of the form of the found regions. We consider simply connected regions in the DSM, all pixels of which have the same height. If more than half of the pixels of the region were marked as high in the previous step, the remaining pixels of the region are also marked as high. Thus, the boundaries of the detected regions are smoothed and false alarms associated with slopes of hills and other sharp differences of the terrain height are rejected. The regions with small area are removed from consideration.
4. Extraction of buildings among the found high regions. Not only buildings and structures, but
also high vegetation (trees) is related to the detected high-rise objects. The normalized difference
vegetation index (NDVI) [
The segmentation procedure splits a satellite image into homogeneous regions; however, buildings can be covered by several regions. The detection of high objects in the DSM allows us to extract an object entirely, but with low accuracy of boundary localization. We suggested the combination of the outputs of high region extraction and segmentation algorithms as follows: if more than 75% of the pixels of the segmented region are marked as high, then the whole region is considered high (the threshold was chosen empirically). This enables to increase the accuracy of object localization and reduce the number of false alarms (for regions of trees closely adjacent to buildings). The result of the proposed approach is shown in figure 1 and in table 1.
The use of convolutional neural networks made it possible to significantly improve the performance of
computer vision algorithms in solving a variety of tasks, including the task of extracting and
classifying objects in satellite images. Due to the increasing interest of various organizations in this
task, at least four global online competitions [
U-Net has a deep convolutional encoder-decoder architecture. At the input, the network receives a
tensor containing satellite image channels and channels with additional data about the scene. At the
output, U-Net returns a single-channel mask coinciding with the size of the original image and
containing the found buildings. The network is trained on batches consisting of fragments with the
size of 112 ×112 pixels randomly taken from the image, to which reflection over the x or y axes is
randomly applied. Its training was performed on a four-channel satellite image of an urban settlement
with the size of about 5 × 5 km and 1-m resolution that contains more than 3,000 objects such as
multi-storey buildings, garages, hangars and cottages. In addition, we used a digital surface model
obtained from the stereo matching algorithm proposed in [
The results of the U-Net network trained on different data sets are shown in figure 1 and in table 1. Building extraction quality of the trained network was also evaluated using satellite images with different spatial resolutions. The results of building extraction from the images with the resolution of 2 m and 3 m per pixel can be observed in the figure 2 and table 2.
a) b) c) d) e) f) g) h) Figure 1. (a) Orthorectified four-channel satellite image (near-infrared, red, green channels) with the 1-m resolution, (b) digital surface model obtained from a stereo pair using a stereo matching algorithm, (c) the hand-drawn ground-truth building regions, (d) detection of buildings in the DSM and refinement of their boundaries using segmentation of spectral information. Building extraction by the U-Net network trained on: (e) image, DSM and NDVI, (f) image, DSM, (g) image and NDVI, (h) only on image. The extracted buildings and the ground-truth building boundaries are displayed in black and red respectively.
The quality of the considered algorithms was evaluated using a fragment of a four-channel satellite image with the size of 900 × 500 pixels and 1-m resolution. The regions marked as buildings by the algorithms were compared with the hand-drawn ground-truth building regions according to the following criteria: 1. Intersection over Union (also known as the Jaccard Index), which is sensitive to the accuracy of localization of the detected objects: = | ∩ |,
| ∪ | where ‒ the regions of the extracted buildings, and ‒ the ground-truth building regions. Its value changes from 0 to 1, where 1 means full overlap of the found regions with the ground-truth ones. 2. The rate and amount of true positives, and false alarms in the output of the algorithms. The results of the comparison can be observed in table 1 and in figure 1.
Of the two approaches to building extraction, the best performance was demonstrated by the U-Net network. Its advantage, as compared with the algorithm that extracts buildings only by using a DSM, is the ability to detect cottages and other low-rise buildings that are not visible in the DSM and these, at least, are one third of the objects in the initial 5 × 5 km image. Also, U-Net performs more accurate detection of the building boundaries.
According to table 1 and figure 1, U-Net shows the good result even without using additional information about the scene. However, the network was trained with height information better detect building boundaries, as was proved by the higher value. The use of the NDVI lead to deterioration in building detection quality. A possible explanation for this might be that NDVI extracts not only vegetation but also buildings. The quality of the building extraction of U-Net was also evaluated on the image of another town with the size of about 5 × 5 km which can be seen in figure A1. Despite the fact that the image differs from the training set by shooting conditions and the composition and type of objects, U-Net (image, dsm) showed for it a similar rate of false alarms, correct and missed detections.
As regards building detection quality of U-Net for satellite images with different resolution, the
network is also able to detect buildings in such cases. Figure 2 illustrates that the quality of building
extraction depends on the building area and the network was unable to detect barely visible objects.
Also, as the resolution is reduced, the quality of detection of building boundaries decreases. Perhaps, it
is possible to improve the accuracy of boundary detection on images with different resolutions by
adding such images to the training set of the network. It should be noted that to date several ways of
improving the U-Net performance have been proposed [
High-rise objects and segmentation U-Net (image, ndvi) U-Net (image, ndvi, dsm) U-Net (image) U-Net (image, dsm)
False Alarms
This paper considered a comparison of two approaches to building extraction from satellite imagery and height data. The best performance was shown by the U-Net-based approach, which extracts a larger number of buildings with a better detection of their boundaries. Furthermore, U-Net shows the good result even without using additional information about scene. However, its application is possible only if there is a sufficiently large data set for training the network. The advantage of the approach based on extraction of buildings from a DSM is the absence of parameters that must be selected for each particular satellite image and it will work on images obtained from different satellites. In the future, it is planned to investigate the effect of the size and composition of the training set on the U-Net building extraction quality as well as the possibility of using trained models of neural networks for satellite images of different areas. 1m 1m, only high buildings 2m 2m, only high buildings 3m 3m, only high buildings
Figure A1. The result of building extraction using the trained U-Net model (image, dsm) for the image with the size of about 5 × 5 km and 1-m resolution. Detected buildings are displayed in green.
This work was carried out within the program of the Ural Branch of the Russian Academy of Sciences № 18-1-1-14.