The paper describes an approach to solving the problem of detection damaged buildings on satellite or other images using convolutional neural networks. To solve the problem, the U-Net convolutional network architecture was chosen. For the study, a proprietary data set containing 50 images with dimensions of 512x512 pixels was used. The application of augmentations was considered to increase the variability of the data set, which made it possible to train the neural network on a small number of images, which had a positive effect on further results. 5 different models of the U-Net architecture were built, the impact of various parameters on the effectiveness of the models was investigated. It has been proven that the initial number of filters has a positive effect on the accuracy of the model. Improved segmentation accuracy for damaged and undamaged buildings. The proposed approach makes it possible to make a preliminary assessment of the degree of damage of buildings and contributes to the implementation of recognition systems based on convolutional neural networks for solving practical tasks.
organizations has its own methodologies that are used to assess the destruction and
calculate the subsequent impact on the economy of the affected regions. For example, the
DaLA methodology [
In general, assessing and overcoming the consequences of emergency situations is a complex and multi-stage process that cannot completely exclude human work. In many cases, expert groups receive information from open sources, internal structures, local administrations and state administration bodies. For a full assessment of the situation, the members of the commissions need to personally study the data from the affected regions, which guarantees accuracy and impartiality in the results. In turn, the automation of the preliminary analysis of the affected regions can make it possible to reduce the time and speed up the examination process, providing information about the most affected areas.
There are many ways to recognize objects in images, in this case – houses. In a number
of studies [
Another study [
The paper [
It is worth noting that the above papers investigate the recognition of damaged
buildings due to natural disasters. In turn, the nature of damage during hostilities can be
significantly different from damage caused by the distaters. Research [
In the study, which examines the assessment of damaged buildings in Kyiv [6] due to the full-scale invasion of Russia, the authors present a completely different possible way of recognizing destroyed objects, namely the use of SAR images with the analysis of satellite image textures and the calculation of the pixel intensity coefficient. The authors argue for the choice of this satellite technology by the fact that weather conditions can become a factor that limits high-quality aerial photography. This study is also distinguished by the fact that the authors show the influence of the size of the building in the corresponding images on the quality and completeness of the classification assessment. Thus, when the area of the building increases, the overall recognition accuracy increases from 64% to 76%.
In recent years, it should be noted an increase in the number of papers devoted to overcoming the consequences of disasters and destruction during hostilities, the authors cite different approaches to solving the problem of recognizing damaged buildings, using both high-quality satellite images and aerial photography from unmanned aerial vehicles. The use of individual methods has its advantages in solving the problem, for example, convolutional neural networks are able to determine various patterns in complex images, and modern architectures are more accurate than classical algorithms.
It should be noted that one of the problems is the need to use a sufficient amount of data for training the model. In turn, it is difficult to obtain high-quality photos from the places of hostilities, so there is a need to use different approaches to increase the variability of the existing data set, for example, data augmentation methods, or optimization of the neural network architecture for working with a small number of images. A combination of different approaches will improve recognition performance.
Problem statement. The purpose of this paper is to study convolutional neural networks for solving the problem of detection of damaged buildings. Building and application of a convolutional network model for detection of damaged buildings on satellite images.
Today, there are a large number of models of different convolutional neural networks. Considering some of the most popular convolutional networks [7-10, 24-26], it is possible to note the achievement of high recognition accuracy on various data sets.
Each architecture has its own advantages in some tasks, for example, the VGG model [7] shows high accuracy in classification tasks, but it is also used in others - objest detection in an image or segmentation. The R-CNN network [9] has been applied to object detection tasks on various data sets and shows reliable results, and the latest modifications, such as Faster R-CNN or Mask R-CNN [11-12] can be used to solve objects search tasks in realtime and image segmentation with higher accuracy than predecessors.
Since this paper deals with the detection of damaged buildings in images, Figure 1 in the next page provides an example of the different computer vision techniques that can be used.
Each technique entails the need to use certain approaches and appropriate models. Since there will be a large number of buildings in the images, the usual classification option, in which one image belongs to one or more classes, is not considered. In turn, the use of object detection, object localization or image segmentation will have both practical and visual significance. Therefore, the authors suggest using the semantic segmentation technique, in which each pixel of the image is associated with one of the classes. This will allow more accurate classification of damaged buildings, generating segmentation maps of affected areas, the results of which will be easier to interpret.
As the selected model, the U-Net convolutional network architecture [8] is proposed, which is based on the idea of a fully convolutional neural network [13], which is effective for capturing both context and spatial information. In principle, this architecture is similar to the encoder-decoder model, has fewer parameters due to the lack of fully connected layers, but compensates for this with the greater complexity of the expanding path. Since this work considers the processing of "static" images, there is no need to use complex models that are designed to solve tasks in real time, which makes the U-Net architecture optimal for the task of detection damaged buildings.
U-Net is built on the basis of a fully convolutional neural network, the authors of the architecture modified it in such a way as to be able to work with a small amount of training data [8], and at the same time obtain sufficiently accurate segmentation of images - which is critical in the field of biomedicine, for research in which and the model was implemented. The basic idea was to supplement the usual sub-network with additional sequential layers, in which the pooling operators were replaced by upsampling layers, increasing the output resolution. In turn, the features of high resolution from the narrowing path are combined with the result of upsampling layers, which allows for more accurate segmentation based on this information.
In this study, to solve the problem of recognition of damaged buildings, the classical architecture of the U-Net network will be used, Figure 2 shows a visual diagram of the network.
Summarizing the work of the model, the structure of the network [8] can be divided into several logical blocks: • • • •
Convolutional Block. It consists of a 3x3 convolutional layer and a ReLU activation function that are repeated twice. This block is used in the Encoder Block as a subsidiary one.
Encoder Block. A Contracting Path block consisting of one Convolutional Block, a 2x2 pooling layer with a step of 2. A Dropout Block is also added to prevent rapid overfitting of the model.
Bottleneck. It is the "narrowest" part of the model, consisting of a Convolutional Block with a maximum multiplier of filters (feature channels). Thus, at this point the model has the lowest image resolution but the highest number of corresponding channels.
Decoder Block. It is an analogue of Encoder Block for Expanding Path, which consists of an upsampling layer, a combination with a corresponding layer from a Contracting Path, a Dropout layer and a corresponding Convolutional Block.
By connecting these blocks together, the U-Net network model will be implemented. Thus, the total number of convolutional layers in the model is 23, taking into account the other layers, the model will contain about 34.5 million parameters for training with an initial value of 64 filters, and 8 million parameters, reducing the base number to 32.
The data set has a direct impact on the operation and results of any neural network. In this paper, the authors use their own data set, created using satellite images from the Google Earth service.
The set has a small amount of data: 50 satellite images with a size of 512x512, which contain images of buildings from the private sector of the city of Mariupol. The pictures were taken in May 2022. At the same time, the data set contains more than 1,500 instances of different buildings, which belong to two classes: damaged ('damaged' label) and undamaged ('normal' label). Figure 3 shows an example of images from the dataset.
It is worth noting that the approach used in the study does not allow to understand the nature of damage inside the building - therefore, when annotating images, only visual damage is used, for example, on the roof or facade of the building. Due to the complexity of annotating buildings of different shapes, as well as taking into account dense buildings in some cases, simple shapes were used when creating the segmentation mask, some parts of the buildings could be combined into one shape.
When annotating images, we used the free Labelme tool, which is publicly available on the Github resource [14]. Figure 4 shows a representation of the dataset, namely the original image enhanced with the OpenCV library and its segmentation map. In general, the data set contains three classes: background – which is responsible for the background image (black color), normal – for annotating undamaged buildings (green), and damaged – for damaged ones (red).
Thus, in total, the dataset contains 100 images: 50 source images to be used to train the model and 50 generated segmentation maps for them, each pixel corresponding to one of the three classes.
Data preprocessing is the next important step for effective model training. Modern studies [15-18] propose various image processing techniques, from classical rotation and mirroring, to the creation of separate procedures for generating images by inserting individual elements [17], or using networks such as GAN. Each approach can be useful in certain situations.
Since the initial data set has a small number of images, the authors decided to apply data augmentation - expanding the set by adding images already existing in the set with certain changes: • • • •
Image shift.
Image scale.
Image rotation.
Image blur.
Before applying augmentations, each image was scaled to the appropriate size specified in the model and normalized accordingly: since pixel values range from 0 to 255, each pixel value in the image was divided by 255. It is worth noting that a large amount of image manipulation can lead to loss of quality and clarity, which will affect the learning of the neural network. That is why various combinations of defined augmentations with limit values for geometric transformations are used in the work. Thus, the probability of image corruption at the pre-processing stage is reduced.
Functions. The standard model uses a softmax function in the output layer. The softmax, or normalized exponential function, transforms a vector of real numbers into a probability distribution of possible outcomes, thus it is suitable for classification tasks, including image segmentation, by transforming some of the pixel values of the original array into an array of class probabilities that sum to 1. Formula 1 displays the softmax activation function [19,22-26]: ( ) = exp( ∑ =1 exp( )
, ) =1 = − ∑ ℎ × log( ( )), where c is a defined class from 1 to k, j is an iteration of classes from 1 to k, output is the output data of the network (image), and exp is the exponent from the corresponding set of values.
In general, the output values of a neural network, without the use of activation functions, are ordinary sets of values that reflect the final result of operations in the layers of the network. In the case of the U-Net architecture, the SoftMax activation function described above is used, which transforms a set of values into a set of probabilities whose sum is equal to 1. This allows you to use these values in the future to calculate the accuracy of the model, its costs and the use of metrics, since the latter uses probability values.
In this paper, the categorical cross-entropy is used as a cost function, which is common in solving semantic segmentation problems (binary in the case of classifying one sample of one class in the image), which measures the 'distance' between the predicted distribution and the actual distribution of classes. Thus, the lower the entropy value, the better the agreement between the prediction and reality, while an increase makes the model worse in terms of predicting the actual results.
Formula 2 shows the application of the function in the case of multi-sample classification [19,22, 23]:
where c is the defined class in the range from 1 to k, output is the output data of the network (image) and ground_truth is the segmentation map (mask) to the corresponding image.
Metrics. To calculate the effectiveness of the model, several metrics are used in this study, namely, the overall accuracy, which reflects the ratio of the number of matching pixels of the original image with the previously created segmentation map, IoU (Intersection over Union) and its modification - MeanIoU, which is respectively the average value of IoU for several classes.
IoU is a popular metric used in machine learning to measure localization accuracy, calculating localization errors in different models. In general, IoU can be represented as the ratio of the common intersection between two areas to their total area. Formula 3 is used to calculate this metric [20,22,23]: =
+ + where c is the class for which the metric is calculated, TP is the 'true positive' value, FP is the 'false positive' value, FN is the 'false negative' pixel value.
Thus, the IoU metric reflects how accurately the model segmented the image according to the reference segmentation map. In turn, the classes of damaged and undamaged buildings can dominate other classes, in the case of this study - the background class, so (1) (2) (3) the authors use the MeanIoU calculation, which reflects the average value of IoU for all three classes, while IoU will be used to calculate the indicator separately for building classes. Formula 4 reflects the calculation of the MeanIoU metric [20,22,23]: (4) where c is the class for which the metric is calculated, IoUc is the calculated IoU value for class c. Thus, using a combination of different metrics, you can get a more accurate picture of the effectiveness of the model both on separate classes and in general.
The models were built in Python using the Tensorflow library and Google's Colab machine learning service, which uses cloud computing to run different pieces of code and train the models [21]. Training was conducted using the NVIDIA T4 GPU graphics processor.
As part of the experiment, an initial model was built: the initial value of the filters was 32, the input image was 256x256 pixels, and the data set was increased by 10 times. The data set was split into training and test sets in the ratio of 80% to 20%, so the model was trained on 400 images for 25 epochs.
Since the model was trained on a small data set and Dropout layers with a firing frequency of 0.3 were applied, the image has typical 'teeth'. During all training epochs, the model improved the results, overtraining was prevented by limiting the number of epochs. segmentation map, for which a test dataset of non-training images was applied.
The overall accuracy of the initial model is 82.28%, but this does not indicate its accuracy with respect to the main classes - damaged buildings. Comparing the IoU scores for each class, the results are as follows: 42.67% accuracy for damaged buildings and 38.37% accuracy for undamaged buildings at the pixel level. To study the results, the authors suggested building models with different parameters (see Table 1).
Based on the results of testing on a separate set, the basic model of the U-Net neural network highlights the general features of buildings and generally correctly segments the image, but individual elements prevent the network from recognizing objects more accurately, for example, the network confuses the courtyard with the roof of buildings, the model also does not recognize objects are small in size well. 5 models were built, including the base model, which had different parameters: the initial number of filters in the model, the number of training epochs, and the applied augmentations. The size of the image could have a positive effect on the quality of the model, but the increase to the original size of 512x512 was not possible for technical reasons. Based on the obtained results, it was possible to improve the overall accuracy of the model to 84.21%, respectively, the recognition of damaged buildings at the level of 45.83% and undamaged, respectively, at 49.14%.
This paper considers an approach to solving the problem of recognizing damaged buildings on satellite or other images using convolutional neural networks. To solve the problem, the U-Net convolutional network architecture was chosen.
For the study, a proprietary data set containing 50 images with dimensions of 512x512 pixels was used. The application of augmentations was considered to increase the variability of the data set, which made it possible to train the neural network on a small number of images, which had a positive effect on further results.
As part of the work, 5 different models of the U-Net architecture were built, the influence of various parameters on the effectiveness of the models was investigated. According to the results, the initial number of filters has a positive effect on the accuracy of the model, in turn, there is no need to train the model on a large number of epochs, since in most cases this leads to overfitting, which can be avoided in case of increasing the dataset with original images. A segmentation accuracy of 45.83% was achieved for damaged buildings and 49.14% for those that were not damaged or could not be visually identified. Thus, the application of convolutional neural networks of the U-Net architecture allows recognition of general features of damaged buildings in images. The proposed approach makes it possible to make a preliminary assessment of the damage degree of damaged buildings and contributes to the implementation of recognition systems based on convolutional neural networks for solving practical problems. [6] Aimaiti, Yusupujiang, Christina Sanon, Magaly Koch, Laurie G. Baise, and Babak Moaveni. War Related Building Damage Assessment in Kyiv, Ukraine, Using Sentinel-1 Radar and Sentinel-2 Optical Images. Remote Sensing. 2022. Vol. 14, 24: 6239. DOI: 10.3390/rs14246239. [7] Karen Simonyan, Andrew Zisserman, Very deep convolutional networks for largescale image recognition, 2015. URL: https://arxiv.org/pdf/1409.1556.pdf. [8] Olaf Ronneberger, Philipp Fischer, Thomas Brox, U-Net: Convolutional Networks for
Biomedical Image Segmentation, 2015. URL: https://arxiv.org/pdf/1505.04597.pdf. [9] Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik. Web Spam Taxonomy, 2014. URL: https://arxiv.org/pdf/1311.2524.pdf. [10] Alex Krizhevskym, Ilya Sutskever, Geoffrey E. Hinton. ImageNet Classification with Depp Convolutional Neural Networks, 2012. URL: https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8 436e924a68c45b-Paper.pdf. [11] Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, Mask R-CNN, 2018. URL: https://arxiv.org/pdf/1703.06870.pdf. [12] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, 2016. URL: https://arxiv.org/pdf/1506.01497.pdf. [13] Jonathan Long, Evan Shelhamer, Trevor Darrell. Fully Convolutional Networks for
Semantic Segmentation. CVPR2015. DOI: 10.48550/arXiv.1411.4038. [14] Labelme. Image Polygonal Annotation with Python. URL: https://github.com/labelmeai/labelme. [15] Khaled Alomar, Halil Ibrahim Aysel, Xiaohao Cai. Data Augmentation in Classification and Segmentation: A Survey and New Strategies. Imaging. 2023. Vol. 9, № 2, p. 46.
DOI: 10.3390/jimaging9020046. [16] Sandhi Wangiyana, Piotr Samczynski, Artur Gromek. Data Augmentation for Building Footprint Segmentation in SAR Images: An Empirical Study. Remote Sens. 2022. Vol. 14, №9. DOI: 10.3390/rs14092012. [17] Sengdeok Bang, Francis Baek, Somin Park, Wontae Kim, Hyoungkwan Kim. Image augmentation to improve construction resource detection using generative adversarial networks, cut-and-paste, and image transformation techniques.
Automation in Construction. 2020. Vol. 115. DOI: 10.1016/j.autcon.2020.103198. [18] Golnaz Ghiasi, Yin Cui, Aravind Srinivas. Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation. CVPR 2021. 2021. DOI: 10.48550/arXiv.2012.07177. [19] Bishop, Christopher M. Pattern Recognition and Machine Learning. Springer, 2006.
ISBN 0-387-31073-8. [20] Aziz Taha, Abdel, Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 2006, 15 (29): 1–28. doi:10.1186/s12880015-0068-x. [21] Colaboratory. Google, 2024. URL: https://research.google.com/colaboratory/faq.html. [22] Peter Bidyuk, Irina Kalinina, Oleksandr Zhebko, Aleksandr. Gozhyj and Tetyana Hannichenko, Classification System Based on Ensemble Methods for Solving Machine Learning Tasks, CEUR- WS. 2023, vol. 3426, pp. 1-11. CEUR-WS.org/Vol3426/paper5.pdf. [23] Irina Kalinina, Peter Bidyuk, Alesandr Gozhyj and Pavel Malchenko, Combining Forecasts Based on Time Series Models in Machine Learning Tasks, CEUR-WS. 2023, vol. 3426, pp. 25-35. CEUR-WS.org/Vol-3426/paper2.pdf. [24] V. Hamolia, V. Melnyk, P. Zhezhnych, and A. Shilinh, (2020). Intrusion detection in computer networks using latent space representation and machine learning. International Journal of Computing, 19(3), 442-448. https://doi.org/10.47839/ijc.19.3.1893 [25] V.Turchenko, E.Chalmers, & A. Luczak, (2019). A deep convolutional auto-encoder with pooling – unpooling layers in caffe. International Journal of Computing, 18(1), 831. https://doi.org/10.47839/ijc.18.1.1270 [26] Y. Bodyanskiy, A.Deineko, V. Skorik, & F.Brodetskyi, (2022). Deep Neural Network with Adaptive Parametric Rectified Linear Units and its Fast Learning. International Journal of Computing, 21(1), 11-18. https://doi.org/10.47839/ijc.21.1.2512