-Accurate detection of vehicles in large amounts of imagery is one of the harder objects' detection tasks as the image resolution can be as high as 16K or sometimes even higher. Difference in vehicles size and their position (direction, they face) is another challenge to overcome to achieve acceptable detection quality. The vehicles can also be partially obstructed, cut off or it may be hard to differentiate between object colour and its foreground. Small size of vehicles in high resolution images complicates the task of accurate detection even more. CNN is one of the most promising methods for image processing, hence, it was decided to use their implementation in YOLO V3. To deal with big high resolution images method for splitting/recombining images and augmenting them was developed. Proposed approach allowed to achieve 81.72% average precision of vehicles detection. Results show practical applicability of such approach for vehicles detection, yet to reach higher accuracy on tractor, off-road and van categories of the vehicles the count in different vehicle categories needs to be balanced, i.e. more examples of the mentioned vehicles are required.
Vehicles’ detection from aerial photography is a very
important and quite a difficult task, especially when it is performed
in real time or high resolution aerial or satellite images are
used for vehicle detection, such as 18000x18000 px. resolution
images in COWC [
In this paper we investigate applicability of Convolutional
Neural Networks. Due to good performance [
Moreover, we split the image into fixed overlapping rectangular frames (a sliding window method). c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Some results show that Yolo V2 performs quite well with
aerial imagery only with applied modifications: "First making
the net shallower to increase its output resolution. Second
changing the net shape to more closer match the aspect ratio
of the data." [
In another vehicles detection solution newer YOLO version
was used [
None of these solutions used splitting and remerging technique with images’ overlapping. They used already presplitted images.
As computing speed is increasing, and technology is advancing neural networks are being optimised, it had been decided to apply best image augmentation/splitting/remerging methods for vehicle detection. In the application of neural network the following set of problems becomes apparent: 1) Having a variety of different resolution images in dataset (HD, Full HD, 2K ...). 2) Uneven vehicles’ sizes in a dataset which are influenced by different ground sample distances (GSD). 3) Uneven vehicles’ count in categories by having more cars than other vehicle categories combined. 4) Almost all of the fully connected convolutional neural networks have a fixed-size first layer and all images should be resized to fit the first layer. 5) Vehicles can be partially obstructed (only part of the vehicle could be seen). 6) Hard to differentiate vehicles from foreground (for example, black car parked in a shadow). 7) Vehicles may be facing multiple directions depending on the camera flight direction and its rotation. 8) Available vehicle detection solutions are limited to detecting a small number of features. 9) After re-merging splitted images, the same vehicle may be detected multiple times.
Currently existing vehicles’ detection solutions are subject
to company trade secrets and companies do not openly discuss
technical specifications and application results (for example
web platform Supervisely [
MAFAT tournament [
Following is the count of dataset images: 1) 1712 images were chosen as training images, about 80% of original training dataset images. 2) After splitting training images into 500x500 pixel pieces, images count rose to 9141. 3) 1986 images were chosen for validation, about 78% of original validation dataset images. 4) 12 227 vehicles were annotated manually by me in the training dataset, Fig. 1. 5) 10 914 vehicles were annotated manually by me in the validation dataset, Fig. 2.
The number of vehicles used in the training images is presented in Fig. 1 and the validation datasets are presented in Fig. 2. The characteristics of dataset images: 1) Images were taken from a variety of locations, some were taken in cities, others in rural areas. 2) Images were taken at a different time of a day. 3) Vehicles were lit from different sides. 4) The resolution of images were different from 900x600 px. to 4010x3668px. 5) Some parts of images were darkened out (for example one half of image was made completely black, while another half of image has picture). 6) GSD (Ground sample distance) of images varied between 5 and 15 cm. 7) Objects in images might have been obstructed by trees or cut off, only part of vehicle might have been seen (for example, a car parked in a garage, a car near the edge of the image).
Couple of images examples taken from dataset Fig 3.
The above dataset was considered sufficient for the evaluation of developed method.
The use of CNNs is complicated due to the dataset having
a variety of different resolution images (HD, Full HD, 2K
...) and uneven vehicles’ sizes in a dataset, see Sect. III.
The different sizes in the images are influenced by different
ground sample distances (GSD) [
One more problem with vehicle detection in images is that the vehicles can be partially obstructed (only part of the vehicle could be seen) for example when car are half parked in garage, or when car are parked alongside tree ant tree branches obstruct car features, or when car is on the edge of image.
As vehicles orientation in images are not constant they may be facing multiple directions depending on the camera flight direction and its rotation. To solve different vehicles orientation problem, the images are augmented with random rotation at 45 intervals Fig. 4.
(a) rotated 0 (original) (b) rotated 45 (c) rotated 90 (d) rotated 135 (e) rotated 180 (f) rotated 225 (g) rotated 270 (h) rotated 315
To increase images’ count, images were augmented by rotating them at 45 degrees intervals. Additionally, dataset images were augmented by flipping them vertically, horizontally and by flipping both horizontally and vertically Fig. 5.
For experiments, convolutional neural network YOLO V3 was used on Darknet framework. YOLO V3 architecture is presented in Fig. 6.
On original YOLO repository the problem was that while
training, detection loss climbed to infinity, when any single
parameter was changed, thus another forked repository [
Fig. 6: YOLO V3 architecture [
Also, it is hard to differentiate an off-road from a car when looking from above, as the body shape of an off-road may differ only slightly (for example, be wider), thus off-road was annotated as a car. Jeep category is hard too, as the only difference between a car and a jeep is that a jeep has a rear spare tire attached or it has a truck bed (like a pickup).
Vehicle categories like cars, jeeps, large vehicles, vans and tractors need to be detected from the aerial photographs and their position needs to be marked by drawing bounding box around each the object. At first, cars’ class had been divided into hatchbacks and sedans, but during manual objects’ annotation it was observed, that if a car is half obstructed and only its front part can be seen, it is impossible to tell whether it is a sedan or a hatchback as the only differentiating factor is the size of rear glass and only the trunk/ boot can be seen. For this reason, sedans and hatchbacks were merged into one vehicle class.
As the dataset contained mostly cars, YOLO learned that if unsure, it should ascribe an object to a car category, that way it could reach better mAP result in a long run than guessing rarer classes. This non-homogeneous dataset problem shows up, if dataset has different number of vehicles for given class in dataset. This non-homogeneous dataset problem could be solved by adding images in which rarer classes’ vehicles are shown or by augmenting a larger number of rearer class images than images with other vehicles.
Cross-validation statistical method was used during YOLO training, the dataset was divided into images for training and validation. The neural network can not see any of the validation images during training, it can only see them when its performance is validated. This method is used to prevent overfitting. The following modifications were performed for the purposes for training and validation images in a dataset: 1) Images’ slicing/ overlapping parameter values modification. 2) Fixing wrongly annotated vehicle data and their bounding boxes’ locations in the datasets. 3) Changing vehicles’ count of classes by adding, merging existing, then reannotating dataset. 4) Choosing images from dataset for training/ validation. 5) Experimenting with images’ manipulations (vertical/ horizontal flipping, image rotation), this drastically improved dataset size. These manipulations were manually coded as YOLO, unlike Tensorflow, does not have these image manipuliations integrated.
The following modifications which were done on YOLO: 1) Change of YOLO layer resolution (mostly first layer, as all images are resized to the same resolution as the first layer size). 2) Experiments with different YOLO configurations and different layers’ count. 3) Change of network parameters (such as anchors, recalculating certain layer size after vehicles’ classes modifications, learning speed). 4) Adding a module to darknet for easier work with split images and for external communication with other programs.
To evaluate performance PASCAL VOC evaluation metrics
were used and the results were compared using AP (average
precision) [
After training the YOLO V3 neural network it managed to detect cars with 78.69% average precision (AP) Fig. 7, large vehicles with 44.85% average precision (AP) Fig. 8. Other vehicle categories such as jeeps, vans and tractors were detected but they were wrongly categorised. That was the reason vehicle category detection average precision was very low. To solve this problem, the dataset needs to have more unified count of vehicles in every category.
Fig. 8: Precision and recall curve for large vehicle category
The above figures show how precision and recall are
corelated, for example, if we choose precision at 95%, then 45%
of cars were detected in validation images at that level of
precision. F-Score [
Fig. 9: Precision and recal graph when all vehicles are merged to one category
This application could be used for statistics (to count how many vehicles are there in a given image), vehicles tracking, prediction of further vehicle movement direction and realtime vehicle detection from real time video feed. A vehicles’ detection application was created so as users could easily configure it and make vehicles’ detection task easier. The user only needs to input images and a couple of parameters to execute vehicles’ detection with CNN.
Results: 1) Dataset was prepared for vehicles detection task by manually annotating all of the vehicles in dataset images. 2) Images’ were augmented to increase dataset size. 3) Method for combining splitting and joining images and using convulutional neural network for vehicles detection was proposed.
4) Proposed method performance was tested by using 1) When YOLO V3 is used together with proposed method is capable of detecting cars with 79% accuracy and large vehicles with 45% accuracy. 2) When proposed method is used, YOLO V3 CNN still has difficulty detecting characteristics of other vehicles, such as off-road, tractors and vans which makes the final detection result lower.
resolution images. 3) Proposed method helps to avoid losing vehicles and their features that would otherwise be lost by resizing high 4) The dataset used for training and validation should have more unified count of vehicles categories (more photos with tractors, large vehicles and jeeps should be added to the dataset).
For future work R-CNN and SSD networks will be trained on Tensorflow framework as those networks are also widely used CNN’s for object detection tasks and they will be tested using same proposed method. Also, as currently used images’ available datasets and photos taken from drones. As the dataset should have more unified count of vehicles categories, more photos with tractors, large vehicles and jeeps should be added to the dataset.