A minimal working version of the computer vision subsystem has been developed specifically for deployment on a research unmanned aerial vehicle (UAV). This subsystem focuses on detecting specific objects present on the surfaces of water bodies and subsequently classifying them. The efectiveness of this subsystem was evaluated by comparing two state-of-the-art models, YOLOv5 and YOLOv8, to determine their suitability for addressing the target problem. To evaluate performance of the resulted model's series of test was performed. It resulted in achieving desired output of object detections but with low accuracy of classification, however such systems can be used as wider-area object detector. According to the obtained results, it can be seen that the system detects objects on the water surface, but the classification of these objects is not good. There are several reasons for this: errors in the labeling of the dataset and the small size of the dataset. A possible scenario of using the built model is the general collection of information about the reservoir without regard to the classification output. In the process of such exploitation, it can be considered as expedient to collect a dataset that will correspond to the data from the drone (the data of the current dataset is data from surveillance cameras and video recordings from boats). In the future, form the dataset according to the developer's requirements, applying the necessary data augmentation steps.
In the modern world of robotics, many tasks require the intervention of artificial intelligence to increase
the number of tasks to be solved [
In the modern period of development of unmanned aerial vehicles comes the realization that many
tasks of research and observation can be transferred to automated drones, which will perform them faster
and better due to the possibility of installing additional computing power as a payload [
Computer vision systems are technologies that give computers the ability to recognize and analyze
visual data [
Unmanned aerial vehicles (UAVs) have various structural elements that determine their functionality and characteristics. The main structural elements of a UAV include: fuselage (body), wings, tail unit, engines, connecting elements, equipment for filming and observation (cameras, sensors). The variety of tasks for the use of UAVs in water and coastal zones illustrate the importance and relevance of the conducted research for various fields of application, including full-scale war on the territory of Ukraine, customs, security, monitoring, ecology and natural science. The purpose of the research is to create a minimum working version of the computer vision subsystem for use on a research UAV and to provide instructions for further improvement of the system and its development.
The feasibility of using AI is due to the fact that there is no clear algorithm for detecting objects on the water surface using image processing methods other than AI. Also, the use of UAVs in combination with AI will allow processing data from large areas of the earth and water surface, which will improve the response to emergency situations with the use of a limited number of human resources [23].
For task of object detection in the image, there are a large number of software solutions that allow you to construct and train a neural network. Examples of such solutions are tensorflow, pytorch, theano, ultralytics, chainer libraries. Since the task of creating a dataset is part of the usual functionality of the libraries, the range of possible options is narrowed to the ultralytics API, which is less flexible in terms of model selection, but provides a wide functionality for working with data. To perform the given task, it is most appropriate to use the Ultralytics API, as they provide the necessary functionality for dataset synthesis and provide interfaces for programming the training of a wide range of models for object detection. The software is written in the Python programming language due to its dynamic typing and automatic garbage collection, as well as a port of the above API for this language.
The subsystem will control the drone, which must move along the route at the points specified by the user, and be able to detect such objects as boats, ships, buoys, garbage islands, swimmers and drowning people from the image from the camera.
The dataset is under development and is a compilation from several data sources: https://universe.roboflow.com/hamdi-ali/plastic-pollution-ugslg, https://universe.roboflow. com/double-o-co-ltd/marine-object-detection-yjybm/dataset/4, https://universe.roboflow.com/ pwnface4-gmail-com/drowning-people. The general principles, application, versatility and features of use are given in [24, 25, 26, 27, 28]
Model definition: since it is planned to implement the model on a Raspberry microcomputer, 2 possible models for training can be distinguished, YOLOv5 due to its small size and YOLOv8 due to the fact that with approximately the same number of parameters as YOLOv5, the model gives a better result, as shown in figure 1.
In the figure 1, the number of parameters is shown on the abscissa axis, and the efectiveness is shown on the ordinate axis.
The following are examples of the considered images. The “Boat” object class is shown in figure 3. The object class “Ship” is shown in figure 4.
The object class “Buoy” is given on figure 5.
The class of objects “Swimmer” is shown in figure 6.
The object class “Drowning man” is shown in figure 7.
The distribution of images by classes is shown in figure 8.
The largest number of markings belongs to the boat class (1629 units). Along with this class, the following classes are well represented (in descending order): swimmer, boat and buoy, respectively 1498, 1270 and 1186. The garbage and drowning man classes contain the least number of images, which can lead to training anomalies. The distribution of images between datasets is shown in figure 9.
This distribution is due to the fact that for the available 6,000 images, the test data set will consist of three hundred images, which is more than enough to test the performance of the model.
The selection of these classes for the dataset was due to the fact that it allows covering a large part of the objects of maritime navigation and interaction. The addition of human images to the dataset is also due to the fact that the deployable drone can be used as a rescue drone and add the functionality of calling rescuers or providing assistance: lifebuoys or vests can be attached to the drone.
For the YOLOv5 model, the confusion matrix is shown in figure 10.
The history of the metrics of the training model is given in figure 11.
We can see that the model classifies swimmers and debris islands well, although this may be the result of insuficient images for these classes. The most successful class for recognition was “boat” with a probability of correct recognition of 52 percent, which is the expected result for this model. The classes “Buoy” and “Drowning man” are recognized worse and usually the model classifies them as background.
This may be due to both their similarity in the image and their small size and few special features. Ships are also poorly recognized, possibly due to the large number of images in the dataset, in which the ship is a tanker on the horizon and, accordingly, has small dimensions in the image.
Figure 12 shows the results of model verification.
The analysis shows that the model recognizes the object correctly, but gives a small percentage of confidence in its predictions. It is because of this fact that some objects remain unrecognizable.
For the YOLOv8 model, figure 13 shows the confusion matrix.
The history of the metrics of the training model is given in figure 14.
Results of model verification is given in figure 15.
The analysis shows that the model recognizes the object correctly, but gives a small percentage of confidence in its predictions. It is because of this fact that some objects remain unrecognizable.
The obtained results give grounds for the conclusion that problems in training the model arise precisely because of the dataset. Accordingly, for further development and obtaining a higher-quality model, an increase in the number and quality of marked images is required.
At the current stage, YOLO8 has better class recognition performance, although both models correctly locate objects, but have low confidence in the obtained results. This could be due to poor annotation, namely the similarity of the classes “Boat” and “Ship”, “Drowning man” and “Swimmer”, so I think it is necessary to add more images, re-evaluate the old ones and add data augmentation to diversify the training data.
After marking another two thousand images, it was decided to move on to training a new model based on the YOLOv8 model. Images with the following types of augmentations were introduced into the dataset: cropping images for better behavior on images with small objects, generating a new image by placing images in a mosaic, which allows the model to better process images with lower resolution and smaller objects, vertical mirroring in order to exclude the possibility of an uneven distribution of boats between the classes with the nose to the left and to the right.
The parameters of the resulting dataset are given in figure 16.
Examples of augmented images of objects are shown in figure 17.
Training was performed over one hundred epochs with a pre-trained model provided by the ultralytics API. In figure 18 shows the history of training metrics of the YOLOv8 model.
Figure 19 shows the confusion matrix for the YOLOv8 model.
The results of detection of the newly trained model are shown in figure 20.
We can see that the results difer to a certain extent, which confirms the correctness of the chosen direction of improving the dataset.
In figure 21a shows the image after processing by the YOLOv8 model with an accuracy threshold of 0.4, which was able to detect the object “Swimmer” at the location of one of the many objects “Boat” with a probability of 0.56.
The image shown in figure 21b has a high probability of recognizing the objects “Boat” and “Ship”, but assigns the recognized objects to the wrong class (misclassification).
A fairly large volume of research was conducted, as a result of which a significant number of results were obtained and summarized, in particular. A single simple object in the frontal image is correctly recognized and classified. The small Buoy object in the background is completely ignored by the model. A small number of relatively large mountain-view objects were classified with high confidence and correctly. The swimmers closest to the camera were most likely detected, all other objects, including the “Boat”, were not detected. When testing the model, the Garbage object was not recognized, and the Boat object was completely ignored. The example of a swimming frame illustrates the correct recognition of several Swimmer objects from a large number of available ones and the complete ignoring of the recognition of Boat objects. For the three Boat objects, the YOLOv8 image model sees two Swimmer objects instead of the expected Boat objects. Non-existent objects are not found, the class of existing objects is confused, possibly due to the small size of the latter.
For the following objects, the accuracy threshold was reduced to 0.1 and the classification function was disabled. Almost all recognized objects were correctly located. The model does not define objects where they do not exist.
The figure 22 shows examples of images with the accuracy threshold reduced to 0.1 and the classification function disabled.
For experimental objects, 100 percent of objects were marked regardless of scale. Marking occurred several times, which can be corrected by filtering the resulting bounding boxes.
In the presence of a large cluster of diverse and diferent objects, large-scale foreground objects were marked. The rest of the objects are consolidated into one large object.
There were options where objects in the foreground only were recognized that were given a large scale, 100 percent of objects were marked regardless of scale, most of the various objects were marked regardless of scale, and non-existent objects were not marked. All obtained results are processed and summarized.
According to the obtained results, it can be seen that the system detects objects on the water surface, but the classification of these objects is not good. There are several reasons for this: errors in the labeling of the dataset and the small size of the dataset.
The comments shown in the figure 23 have been received from the API developers for working with artificial intelligence.
A possible scenario of using the built model is the general collection of information about the reservoir without regard to the classification output. In the process of such exploitation, it can be considered as expedient to collect a dataset that will correspond to the data from the drone (the data of the current dataset is data from surveillance cameras and video recordings from boats). In the future, form the dataset according to the developer’s requirements, applying the necessary data augmentation steps.
The idea of writing the article belongs to all authors. Viktorija Smolij built and trained the model, Natan Smolij performed testing and analyzed the training results, Sergii Sayapin collected and marked up the dataset, performed the design of the materials.
When writing the article, special thanks should be expressed to the head of the department Oleksandr Rolik, Volodymyr Oliynyk, Yury Berdnyk and Mykola Shynkevych. Great gratitude for the explanation and organization of work, support and kindness.