In this article, a convolutional neural network model is presented for semantic segmentation of aerial images in urban areas using the VGG16 architecture as the encoder and UNet as the decoder. The model was trained and evaluated using the publicly available dataset named Semantic Drone, which consists of aerial images acquired at altitudes ranging from 5 to 30 meters. Various data augmentation techniques, such as random elastic deformation and brightness adjustment, were applied to enhance the model's generalization capability. The obtained results show an average accuracy of 81% in segmenting 23 diferent classes, including people, cars, and dogs. Additionally, an inference speed of up to 50 fps was achieved after optimizing the model on a GPU. Overall, the proposed model has the potential to be employed in drone security applications and decision-making processes in urban areas.
CEUR ceur-ws.org
In recent years, there has been significant interest in the application of deep learning techniques
for semantic segmentation of aerial images [
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings
In the scientific literature, diferent strategies have been proposed to address challenges
associated with deep learning-based semantic segmentation in aerial images. For example,
new techniques have been developed to generate synthetic data and augment the training
dataset [
This work presents the evaluation of a convolutional neural network applied to the task of semantic segmentation of aerial images using the VGG16 and U-Net architectures with a publicly available dataset for training and validation. The remainder of the document is structured as follows: in Section 2, the selected architecture for training the semantic segmentation model, the dataset used, and data augmentation techniques are detailed. Section 3 describes in detail the selected parameters for the training stage of the model. Section 4 presents the experimental results of our work, focusing on evaluating the accuracy of the deep learning-based semantic segmentation model for aerial images on a public dataset for each of the classes and the optimization required to run the model in real-time. Finally, Section 5 discusses the conclusions of the work and presents possible future directions for research in this field.
In this section, we will describe in detail an approach to semantic segmentation based on a
neural network architecture that combines an encoder based on the VGG16 architecture [20]
and a decoder based on the U-Net architecture [
The VGG16 architecture [20] represents a prominent convolutional neural network with deep significance in image classification tasks. Its structure is characterized by a sequence of convolutional and pooling layers, followed by fully connected layers at the top of the network. In the task of semantic segmentation, the VGG16 network plays a fundamental role as an encoder capable of extracting highly relevant features from input images. This architecture has earned a prominent place in the fields of computer vision and deep learning due to its ability to understand and represent complex features in images, making it valuable in a variety of applications.
On the other hand, the U-Net architecture [
By combining both architectures into a single architecture, see Table 1, the inherent strengths of VGG16 as an encoder and U-Net as a decoder are leveraged, resulting in a highly efective approach for semantic segmentation of images. VGG16, with its deep structure of convolutional and pooling layers, excels at extracting visually relevant features from input images, identifying patterns, textures, and key details. These features, acting as high-level knowledge, are essential for semantic segmentation. On the other hand, U-Net, with its specific encoder-decoder design, specializes in the precise reconstruction of segmentation masks. The downward section of U-Net simplifies the task by reducing spatial resolution, while the upward section recovers fine details and local context. The key to this combination lies in the seamless transition between both architectures, using features extracted by VGG16 as input for the upward section of U-Net. This approach provides accurate and consistent segmentation by combining rich detail information and local context with high-level features.
In the proposed architecture, the VGG16-based encoder is employed to extract features from the input aerial images, which are then fed into the U-Net-based decoder to generate the ifnal segmentation mask. Additionally, regularization techniques such as dropout and batch normalization are utilized to enhance generalization and prevent overfitting.
The categorization of aerial images by altitude is a key approach in semantic segmentation, as the features and objects present in the images vary significantly depending on the altitude at which the image was captured.
These images can be classified into three main categories based on their acquisition altitude: low, medium, and high. Low-altitude images are typically captured at heights of less than 30 meters and show fine details of objects such as buildings, vehicles, and pedestrians. Mediumaltitude images are obtained at altitudes between 30 and 150 meters, providing a broader view of the photographed area, allowing for a better understanding of the context and distribution of objects in a scene. On the other hand, high-altitude images are taken at altitudes above 150 meters and ofer an overview of a region, facilitating the understanding of the distribution of objects on a large scale.
This categorization by altitude enables semantic segmentation models to focus on specific features of the images that are relevant to the corresponding acquisition altitude. This can significantly improve the accuracy and eficiency of semantic segmentation models, especially when deep learning techniques are employed.
The dataset used in this work is the Semantic Drone Dataset [21], which focuses on the semantic understanding of urban scenes to enhance the safety of autonomous drone flight and landing procedures. The dataset’s images depict more than 20 houses from a top-down (bird’s-eye) view acquired at an altitude of 5 to 30 meters above the ground. A high-resolution camera was used to capture images of size 6000 × 4000px (24Mpx). This dataset includes labels for 24 diferent classes; Table 2 displays the 24 classes and their assigned RGB values. Figure 1 shows four examples of RGB images and the various represented scenarios. Figure 2 displays the corresponding masks for the example images, where diferent colors represent the classes present in the dataset. The training set consists of 400 publicly available images, while the test set comprises 200 private images. This dataset is widely used in research on deep learning-based semantic segmentation for aerial images due to the diversity of objects and urban contexts presented in the images, posing an interesting challenge for machine learning models.
Generating synthetic data or applying data augmentation techniques is a common approach to enhance the ability of semantic segmentation models to generalize and adapt to diferent scenarios and conditions. Data augmentation involves creating new images from the original ones by applying random transformations, such as rotation, scaling, brightness changes, contrast adjustments, among others.
A common data augmentation technique used in semantic segmentation is called ”elastic deformation-based data augmentation,” which involves applying a random elastic deformation to the original image, creating a new synthetic image. This is achieved by adding a small fraction of a random elastic displacement field to the original position of each pixel. The random elastic displacement field is generated by a white noise function that is turned into a vector field
Where ′ is the position of a pixel in a deformed image, is the original position of the pixel, and is the displacement of the pixel’s position. The formula indicates that the deformed position of the pixel is equal to its original position plus the displacement, along with the (1) contribution of the partial derivatives of the displacement function in the and directions.
This technique is used to simulate deformations that may occur in an aerial image due to factors such as lens distortion, aerial vehicle movement, among others.
Another popular data augmentation technique is random cropping, which involves cutting a random portion of the original image and using it as a new image. This technique is particularly useful for creating synthetic images containing partially visible objects, which can help the semantic segmentation model learn to recognize objects in challenging conditions.
Equation 2 represents the random cropping function used, = random(0, − ); = random(0, ℎ − ) (2)
Where and ℎ are the width and height of the original image, respectively, and and are the width and height of the desired crop, random() represents a function that returns a random number within the specified range.
The data augmentation technique by brightness adjustment involves adjusting the brightness of images to enhance the model’s ability to generalize and handle diferent lighting conditions. This technique involves adding a constant value to all pixels in the image, which increases or decreases brightness. The formula used for brightness adjustment can be seen in Equation 3. = + (3)
Where is the original image, ℎ is the image with increased brightness, and is the constant value added to each pixel. The value of can be randomly generated within a specific range to create variations in the image’s brightness.
All images used during the training stage undergo these data augmentation techniques to increase the size of the dataset. Figure 3 shows two results of data augmentation, and Figure 4 shows the corresponding masks after being processed with the same techniques.
Combining diferent data augmentation techniques can significantly increase the diversity and quantity of data available for model training, improving its accuracy and ability to generalize to new scenarios and conditions. However, it is important to note that excessive data augmentation can also result in overfitting the model to the training data, limiting its ability to generalize to new data.
The training process of the machine learning-based segmentation model requires certain resources and tools. One of the most crucial elements is hardware, as computational resources are needed to efectively carry out the model training. In other words, a computer with suficient processing power is required to perform the necessary mathematical operations for training the model.
Additionally, it is necessary to define training hyperparameters, which are variables that control the model training process. These hyperparameters may include batch size, learning rate, number of epochs, among others. Proper selection of hyperparameters can have a significant impact on the model’s performance.
To conduct the training of the neural network used in this study, a computer equipped with a Windows 11 operating system, an Intel Core i7 processor, an NVIDIA GeForce RTX 2060 graphics card, and 32 GB of RAM was employed. This type of hardware configuration is commonly used in deep learning tasks due to its high processing capability and available memory.
The RTX 2060 graphics card, chosen for this study, plays a crucial role in the training process of neural networks thanks to its Tensor core architecture. This feature enables substantial acceleration in matrix operation calculations, an essential function in image processing, particularly in applications like semantic segmentation.
The extensive RAM capacity available in the computer system is an essential resource that allows eficient storage and processing of large datasets, such as the one used in this study. This enhanced capacity significantly facilitates the neural network training process and ultimately reduces the processing time required to achieve a well-performing trained model.
For the training of the neural network, various hyperparameters were utilized that influence the performance and accuracy of the semantic segmentation model for aerial images.
Firstly, an input size of 256 × 256 pixels was used for each image. This size was chosen to balance model accuracy with the time and resources required for training.
The batch size, referring to the number of images used in each iteration during training, was set to 8. This choice was based on the available memory capacity in the hardware used for neural network training.
The number of epochs was fixed at 200, meaning the training dataset was iterated 200 times to adjust the neural network weights and optimize the model.
The initial learning rate was set to 100 − 6 , indicating the rate at which the neural network weights are updated during training. This value was adjusted to ensure an appropriate convergence rate of the model.
Lastly, a Dice coeficient function was used for calculating the stochastic gradient descent in the optimization process. This function is commonly employed as a metric for evaluating semantic segmentation models, allowing the comparison of the overlap area between the predicted segmentation mask and the ground truth segmentation mask.
In order to assess the performance of the trained model, two widely used metrics in the literature are proposed: the Dice coeficient and the Jaccard coeficient or Intersection over Union. The Dice coeficient is a similarity metric employed in semantic segmentation tasks, such as deep learning-based aerial image segmentation. It is commonly used as a loss function during model training and can also be used to evaluate the quality of segmentation on the test set.
The Dice coeficient is calculated as the ratio between the area of intersection between the segmentation mask generated by the model and the true mask of the image, and the total area of the combined two masks. The Dice coeficient value ranges from 0 to 1, where a value of 1 indicates perfect segmentation—meaning the model-generated mask precisely matches the real mask of the image. A value of 0, on the other hand, indicates that the segmentation performed by the model is completely incorrect.
The Dice coeficient can be calculated using the following mathematical formula: Dice = 2 × | ∩ | | | + | | (4)
Here, and are the masks of the segmentation generated by the model and the actual segmentation of the image, respectively. The symbol ∩ denotes the intersection operation between two sets, and | | and | | represent the size of sets and , respectively.
The Jaccard Coeficient is a commonly used metric to assess the similarity between two datasets. In the context of semantic segmentation of aerial images, the Jaccard Coeficient can be employed to measure the accuracy of the segmentation obtained by the neural network.
The formula for the Jaccard Coeficient is expressed as the ratio between the intersection of two sets and their union, and is defined as follows: (, ) = | ∩ | | ∪ | (5)
Here, A and B are the sets being compared, ∩ is their intersection (i.e., the elements common to both sets), and ∪ is their union (i.e., all elements appearing in at least one of the sets).
To assess the model’s performance, a cross-validation was conducted using 100 images from the public dataset that were not part of the training set. Figure 5 presents examples of input images and predicted masks compared to ground truth.
The results shown in Table 3 indicate a relatively high average accuracy in semantic segmentation of aerial images using the proposed neural network. The average accuracy across all classes was 0.791, suggesting that the model can correctly identify most objects in the image.
Looking at the results by class, segmentation of objects like pools, persons, dogs, and bicycles had quite high accuracy, surpassing 0.8 in each case. On the other hand, objects like fences, obstacles, and dirt areas had lower accuracy, possibly due to the dificulty of distinguishing these objects from their surroundings.
It is also observed that the VGG16 and UNet-based neural network achieved a high Dice coefifcient for classes of persons, cars, and dogs, with values of 0.606, 0.940, and 0.978, respectively. These classes are of vital importance in security monitoring and trafic management in urban areas.
Overall, the obtained results are promising and suggest that deep learning-based semantic segmentation can be a useful tool in applications requiring detailed understanding of aerial images, such as security surveillance, urban planning, and precision agriculture. However, it is essential to note that the model’s accuracy can be influenced by various factors, including image quality, scene complexity, and variability in detected objects.
An evaluation of the model’s inference time was conducted using diferent hardware configurations, including CPU and GPU. The results show that when running on CPU, an inference rate of 2 fps was achieved, which is quite low for practical real-time applications. On the other hand, when evaluating the model using the GPU without optimization, an inference rate of 20 fps was obtained, representing a significant improvement compared to the CPU. However, by implementing the optimized model on the GPU, an average inference rate of 50 fps was achieved, demonstrating the importance of optimization to enhance model performance. In general, these results suggest that the implementation of deep learning-based models for real-time semantic segmentation of aerial images is feasible using suitable hardware and optimization techniques.
Inference time is a critical factor in real-time applications, such as the monitoring and analysis of aerial images. In our study, the model’s performance in terms of processing speed was evaluated using diferent hardware configurations. When performing inference on a CPU, the model took an average of 0.5 seconds to process each image, resulting in a frames-per-second (FPS) rate of 2. Implementing the model without optimization on the GPU increased the FPS rate to 20. However, with the implementation of optimization techniques, such as operation fusion and precision conversion, an average FPS rate of 50 on the GPU was achieved. This means that the model could process 50 images per second, highlighting the importance of optimization in improving model performance.
In summary, this article has proposed a robust solution for semantic segmentation in aerial imagery, leveraging a neural network architecture amalgamating VGG16 and UNet. The Semantic Drone Dataset served as the cornerstone for training, and the integration of data augmentation techniques further amplified model accuracy.
The achieved results are indeed promising, showcasing a Dice coeficient surpassing 50% for the majority of classes. Noteworthy enhancements in model accuracy were realized through the judicious application of data augmentation and hyperparameter optimization.
In terms of inference time, a substantial boost was evident with GPU utilization and model optimization. This translates to a more streamlined application of the model in real-time scenarios.
The demonstrated eficacy of deep learning-based semantic segmentation opens avenues for improved security and eficiency in diverse domains such as smart city planning and monitoring, precision agriculture, environmental surveillance, and other drone-centric applications.
The authors extend their gratitude to CONACYT (National Council of Science and Technology, Mexico) for their support in facilitating this research. [19] R. Ratajczak, C. F. Crispim-Junior, E. Faure, B. Fervers, L. Tougne, Automatic land cover reconstruction from historical aerial images: An evaluation of features extraction and classification algorithms, IEEE Transactions on Image Processing 28 (2019) 3357–3371.
URL: https://ieeexplore.ieee.org/document/8630683/. doi:10.1109/TIP.2019.2896492. [20] S. Liu, W. Deng, Very deep convolutional neural network based image classification using small training sample size, IEEE, 2015, pp. 730–734. URL: http://ieeexplore.ieee.org/ document/7486599/. doi:10.1109/ACPR.2015.7486599. [21] T. U. Graz, Semantic drone dataset, 2023. URL: https://www.tugraz.at/index.php?id=22387.