The development of robust computer vision models is frequently constrained by the scarcity of precisely annotated datasets, high labeling costs, and strict regulatory limitations on data collection. Real-world datasets often lack diversity regarding critical edge cases and night scenarios, thereby limiting model reliability. To address these challenges, this paper proposes an iterative approach to generating synthetic datasets using the Unity engine, specifically tailored for vehicle detection tasks. We developed and evaluated three generations of synthetic datasets, systematically reducing the Domain Gaps. The performance of YOLOv8 models trained on these datasets was evaluated against the VisDrone real-world baseline using Synthetic-only and Fine-tuning strategies. Experimental results demonstrate that while pure synthetic data yields lower mAP compared to large-scale real datasets, it achieves superior F1-scores at high confidence thresholds. Crucially, fine-tuning experiments reveal that improving synthetic data quality significantly enhances data eficiency. A model pre-trained on the best iteration synthetic dataset and fine-tuned on just 50 real images achieved accuracy comparable to a model trained on earlier synthetic iterations requiring 400 real images. These findings suggest that high-quality synthetic data serves as a critical force multiplier in data-scarce environments, enabling the rapid and cost-efective deployment of detection systems.
The application of computer vision tasks is experiencing a substantial surge in popularity. However,
developing robust models is often hindered by a lack of available and precisely annotated datasets.
The process of data collection and manual labeling is not only expensive and labor-intensive, but also
frequently leads to annotation inaccuracies [
In this context, the importance of synthetic data cannot be overstated. It facilitates the scaling of
datasets, provides precise control over rare and critical scenarios, and resolves privacy concerns. Using
synthetic data considerably reduces costs and accelerates the development process thanks to automatic
labeling and rapid iteration [
The rapid development of Unmanned Aerial Vehicles (UAVs) has expanded their applications from simple
monitoring to complex autonomous operations. Key tasks, including automatic landing systems [
Recent research in aerial imaging has demonstrated the potential of synthetic data for a wide range
of tasks [21, 22]. These applications range from drone detection using purely synthetic datasets
and semantic segmentation of the environment from a UAV perspective to the creation of complex
multi-modal datasets for urban environments [
This study focuses on the use of synthetic data to solve complex computer vision problems for UAVs applications. The main objectives are to determine the optimal training strategy for neural networks using synthetic data and to evaluate the comparative efectiveness of diferent data generation approaches.
The VisDrone dataset, a comprehensive collection of images captured by UAV, was chosen as a baseline for object detection tasks. Although the standard VisDrone 2019-DET dataset contains 10 object classes, preliminary training with the YOLO v8 model indicated that the "Car” class exhibits the highest instance representativeness and the best F1 score performance, see Figure 1. Consequently, this study focuses on single class detection for the "Car"’ category. The dataset was filtered to include only images containing this specific class, resulting in the following distribution: 5244 images for training, 652 for validation, and 1005 for testing.
The experimental task was defined as a single-class detection of the "Car” category. In accordance with the oficial VisDrone evaluation protocol, special ignore regions, labeled as class 0, were retained. Predictions overlapping these regions were excluded from the scoring process to ensure the accuracy of metric calculations. The YOLOv8n model was used for all comparative experiments, training parameters were set to 100 epochs with an input image size of 640x640 pixels.
Three diferent training modes were evaluated: • Real data (The model was trained exclusively on the real VisDrone dataset) • Synthetic data (The models were trained exclusively on generated synthetic data) • True Positives (TP), False Positives (FP), and False Negatives (FN) • Precision, Recall, and F1-score • Mean Average Precision (mAP) ranging from IoU 0.1 to 0.95
The baseline model trained on the real dataset achieved mAP@0.5 of 0.6723. The dependence of mAP on IoU and F1 score on Confidence for this baseline network are presented in Figure 2.
To generate the synthetic datasets, virtual environments were divided into individual scenes representing diferent locations, environmental conditions, and day/night cycles. The Unity Perception Package was utilized to automatically generate precise annotations for objects in each frame. To achieve a high degree of realism, the generation pipeline incorporated an integrated system for weather, lighting, and sky simulation, which allows for specific date and time settings to replicate real-world environmental conditions. Furthermore, the scenes were complemented by a trafic system, simulating the behavior of both vehicles and pedestrians. Finally, the High Definition Render Pipeline (HDRP) was employed for advanced post-processing to enhance the visual fidelity of the generated images.
The development process was structured around short iterative cycles, consisting of dataset generation, model testing, and subsequent refinement of the generation pipeline. The experimental phase included three distinct iterations: • Iteration 1, consisted of 6282 images (640x640), utilizing 34 3D car models and 15 other transport models • Iteration 2, expanded to 9158 images with increased resolution (1400x788), utilizing 63 car models and 17 other transport models • Iteration 3, comprised 6289 images (1400x788), utilizing 84 car models and 17 other transport models
The first iteration served as a rapid test designed to identify the fundamental challenges of the synthetic approach. Testing revealed initially low performance metrics, with the synthetic model achieving an mAP@0.50 of 0.2367, compared to 0.6723 for the baseline real-data model. However, as illustrated in Figure 4, the primary performance discrepancy was observed in the Recall metric, while Precision remained relatively comparable to the baseline.
Detailed analysis identified several critical detection issues. First, the camera angle in synthetic scenes was often directed vertically downward, which difered from the intended UAV perspective. Second, there was a significant domain gap regarding vehicle colors; while the synthetic dataset utilized high-contrast colors (white, black, red, blue), most real-world vehicles appeared in shades of gray, dark brown, or black. Additionally, the model struggled with detection in chaotic parking arrangements, identifying small objects from high altitudes, and recognizing local specific vehicles such as taxis and police cars, which possessed distinct color schemes and models not present in the training set.
In the second iteration, the image resolution was adjusted to match the VisDrone standard, and the dataset size was increased. Extensive improvements were made to bridge domain gaps. To bridge the appearance gap, models of local transport were added, the vehicle color palette was expanded to include missing color tones, and realistic headlight illumination was implemented for night scenes. To address the content gap, the environment was enriched with parking lots, new locations featuring trees and occlusions, and Bezier paths were implemented to simulate realistic drone flight patterns across three diferent camera angle ranges.
Testing of the second dataset demonstrated improved results. The mAP@0.50 rose to 0.402, compared to 0.2367 in the first iteration and 0.6723 as baseline. As shown in Figure 5, while Recall improved, it remained the primary limiting factor.
Despite the improvements, new detection challenges emerged. A serious issue was the discrepancy between amodal and modal annotations. Unity annotates only the visible portion of a vehicle behind occlusion (modal), while VisDrone’s annotations cover the entire object (amodal). This discrepancy resulted in the model simultaneously producing both false positives and false negatives due to bounding box mismatches. Besides, the model struggled with vehicles with sunroofs or panoramic roofs, as well as a lack of examples with vehicles occluded by trees or with multiple occlusions in trafic jams. A major problem was the limitation of rendering high-density trafic at night due to real-time performance limitations.
Before the third iteration, an analysis of the second synthetic dataset was performed by splitting it into individual scenes and training small neural networks on each scene separately. This testing revealed that increasing the number of images without diversifying locations and camera angles yielded diminishing returns. Specifically, it was found that 700-1000 images per location are suficient for our conditions, generating data beyond this threshold without adding diversity does not improve performance. Analysis also identified that certain scenes in the second iteration were inefective. Consequently, for the third dataset, 2550 images were retained from the second iteration, while four scenes were redesigned, and 3739 newly generated images were added. A primary challenge involved night scenes. While realism was considerably improved in the second iteration, the Unity engine imposes real-time limits on the number of light sources per tile. Simulating night scenes with high object density and trafic jams required a compromise between lighting realism and performance, which ultimately yielded positive results.
Testing of the third dataset demonstrated further improvements. The mAP@0.50 reached 0.4915, compared to 0.402 in iteration 2, 0.2367 in iteration 1, and real base 0.6723. Notably, as illustrated in Figure 6, at high confidence thresholds, the F1-score of the synthetic network outperforms that of the real-data model. This suggests that synthetic annotations can be extremely useful in tasks requiring high detection confidence.
A consistent performance increase was observed with each iteration, but the rate of improvement notably slowed, as shown in Figure 7. Conversely, the efort and resources required to generate each subsequent iteration increased substantially. It remains challenging to create synthetic data that fully outperforms or matches a large real-world dataset across all metrics. On the other hand, synthetic data of varying quality levels may be suitable for diferent tasks. Figure 8 shows detection results on the same test frame using networks from diferent iterations. Crucially, in specific operational ranges, such as high confidence detection (Confidence > 0.8), high-quality pure synthetic data can outperform models trained on manually labeled real data, as illustrated in Figure 7.
This section examines the efectiveness of knowledge transfer from synthetic models to real-world domains using small subsets of real data. Experiments focused on methods for selecting real-world images for fine-tuning and analyzing the impact of synthetic data quality on the size of the required real-world dataset.
To determine the optimal strategy for selecting real data, we compared two sampling methods using a subset of 200 images from the real dataset. The first set (200Hard), consisted of the 200 most dificult frames where the purely synthetic model failed. The second set (200Random), was a uniform random selection proportional to all scenes in the training set. Experimental results indicated that a uniform random distribution is superior for fine-tuning, as illustrated in Figure 9. Based on this finding, we generated randomized subsets of 50, 100, 200, and 400 real images for subsequent experiments.
Following extensive experimentation with hyperparameter tuning, a two-stage training protocol was established. In the first stage, the first 10 layers of the network are frozen, and the model is trained for 20 epochs with an initial learning rate of lr0=0.01 and a final learning rate factor of lrf=0.01. In the second stage, all layers are unfrozen, and training continues for an additional 40 epochs with a reduced initial learning rate of lr0=0.001 and a final factor of lrf=0.1. The fine-tuning results for networks pre-trained on synthetic datasets (Iterations 1, 2, and 3) using varying amounts of real data are presented in Table 1.
The experimental results highlight a substantial improvement in data processing eficiency driven by the quality of synthetic data. Specifically, the model pre-trained on the high-quality synthetic dataset (Iteration 3) and refined with only 50 real images achieved performance metrics comparable to the weaker model from Iteration 1 refined with 400 real images (0.5636 versus 0.5722). This indicates that improvements in synthetic data generation reduced the need for annotated real-world data by nearly 8 times to achieve comparable accuracy. Additionally, it was observed that high-quality synthetic data reaches a performance saturation point much earlier.
In relatively simple object detection tasks, synthetic datasets can produce results close to those achieved
with real data [
Ultimately, the addition of a small amount of real data significantly boosts performance metrics. High-quality synthetic data proves to be most critical specifically in conditions of severe data scarcity, acting as a powerful multiplier for limited real-world annotations.
The authors have not employed any Generative AI tools. arXiv preprint arXiv:1810.08705 (2018). doi:10.48550/ARXIV.1810.08705. [17] K. Konen, T. Hecking, Increased robustness of object detection on aerial image datasets using simulated imagery, in: Proceedings of the 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), IEEE, Laguna Hills, CA, USA, 2021, pp. 1–8. [18] Q. Li, L. Mou, Q. Xu, Y. Zhang, X. X. Zhu, R3-net: A deep network for multi-oriented vehicle detection in aerial images and videos, IEEE Transactions on Geoscience and Remote Sensing 57 (2019) 5028–5042. doi:10.1109/TGRS.2019.2895697. [19] M. Krump, P. Stütz, Deep learning based vehicle detection on real and synthetic aerial images: Training data composition and statistical influence analysis, Sensors 23 (2023) 3769. doi: 10.3390/ s23073769. [20] N. Morrical, J. Tremblay, Y. Lin, S. Tyree, S. Birchfield, V. Pascucci, I. Wald, Nvisii: A scriptable tool for photorealistic image generation, arXiv preprint arXiv:2105.13962 (2021) 9. [21] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. Gupta, A. Farhadi, Ai2-thor: An interactive 3d environment for visual ai, arXiv preprint arXiv:1712.05474 (2019). [22] T. Nikitina, B. Kuznetsov, N. Ruzhentsev, O. Havrylenko, V. Volosyuk, O. Shmatko, A. Popov, S. Zhyla, V. Pavlikov, E. Tserne, Y. Averyanova, O. Sushchenko, M. Zaliskyi, O. Solomentsev, I. Ostroumov, N. Kuzmenko, Algorithm of robust control for multi-stand rolling mill strip based on stochastic multi-swarm multi-agent optimization, in: S. Shukla, H. Sayama, J. V. Kureethara, D. K. Mishra (Eds.), Data Science and Security, Springer Nature Singapore, Singapore, 2024, pp. 247–255. [23] G. Rizzoli, F. Barbato, M. Caligiuri, P. Zanuttigh, Syndrone - multi-modal uav dataset for urban scenarios, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023. doi:10.48550/arXiv.2308.10491. [24] H. Lechgar, H. Bekkar, H. Rhinane, Detection of cities vehicle fleet using yolo v2 and aerial images, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 42 (2019) 121–126. doi:10.5194/ isprs-archives-XLII-4-W12-121-2019.