Dynamic acquisition of an image in three-dimensional space in dynamic mode with its subsequent processing in recognizing structural objects of the exact nature is an urgent task. It needs to ensure the high accuracy of the recognition result and the correct, complete definition of an image. Further calculation of the number of such objects is required. Moreover, it is essential to ensure detection functions inside such a self-organized system in case of classifying input data. In this work, we propose a novel self-organized automated system in which one or several UAVs are controlled and monitored to acquire images of detected objects, considering one object at a time. The outcome of this study serves as the basis for the creation of new means that can launch and monitor unmanned aerial vehicles over subsets of the studied space area according to the given initial data. The architecture designed in this way allows for achieving the appropriate level of organization when determining the next steps in functioning subsystems and components. The conducted experiments confirm the possibility of practical implementation of the proposed architectural solutions.
Obtaining an image in three-dimensional space in a dynamic mode with its further processing in
the context of structural object recognition is an urgent task, as it requires ensuring not only the high
accuracy of the recognition result but, first and foremost, ensuring the correct, complete definition of
the image. Aspects of creating automated systems are considered in works [
To resolve the mentioned above problem, physical devices, and the control system must perform
integration functions. Such a management system should be able to provide the necessary interfaces
for third-party solutions, with the help of which tasks will be received for a group of devices.
Moreover, it is essential to ensure detection functions inside such a self-organized system in case of
classifying input data [
The structure of the paper is as follows. Section 2 presents an analysis of existing solutions to manage UAVs. Section 3 details the proposed architecture of a self-organized automated system for dynamic image acquisition of structural objects in three-dimensional space. Moreover, it also describes a novel approach to controlling and monitoring the operational missions of a group of UAs. Section 4 reveals the experimental results performed by the proposed approach. Section 5 concludes the conducted research and gives hints for further investigations.
The operations of a group of UAVs are similar to those used to control a single UAV. However,
managing a group of UAVs in real-time requires providing more objects in a system and calculating
their relative location [
The tasks of obtaining images of structural objects in three-dimensional space, considering the planned means, require developing a system that would combine means with different functional purposes. The specifics of these tasks also depend on using and managing territorially distributed resources [13]. Communication support is conducted with the involvement of appropriate information and communication means. An essential element of such systems is ensuring control of all components through implementing automation. This requirement comes from the orientation towards the use of the system by domestic users, as was suggested in [14]. Such objects were volumetric areas in the studied part of three-dimensional space, where images were obtained.
The management level in the automated system highlights the process of forming a coordinated space in the working environment, such as an orchard, and the selection of permitted and prohibited areas for UAV flight. Some researchers [15] integrate the flight data management subsystem into an automated system based on DJI GS PRO technology [16]. Applying DJI GS PRO technology results in a set of raw data presented in the form of three-dimensional coordinates [17] of the Global Positioning System (GPS). The primary centralized subsystem takes this data as input parameters of the subsystem to transform the coordinates in the matrix of states.
Positioning of devices in three-dimensional space is carried out using navigation systems, such as GPS. Such systems ensure the fairly accurate positioning of devices in three-dimensional space, as in work [18]. At the same time, a simple set of coordinates cannot fulfill the necessary tasks. To solve the problem, where a group of UAVs must perform recognition of structural objects in real-time, other, more accurate means were proposed. For example, the Real Time Kinematic (RTK) [19] technology was utilized to receive input data for building routes and creating an automated UAV group control system [20, 21]. Real-time kinematics ensured obtaining the plan coordinates and heights of points of the experimental environment to centimeter accuracy using a satellite navigation system. The process of obtaining coordinates and their integration into the structural object detection subsystem predisposes the development of an automated UAV group control system.
According to the analysis of the subject domain, the requirements for the automated control system of several UAVs, according to the functions of the system and its characteristics, the architecture of the developed system was synthesized in the form of a set of several components: the formation of a group of UAVs, a “smart” route planning system, a self-healing system, a work planning system, and system management and monitoring. The result of the automated system is a combined video series, which is then fed to the module for recognition and calculation of the number of structural objects. The combination of the components mentioned above of the monitoring process by a group of UAVs for object detection is the basis of the architecture of the proposed self-organized automated system.
This section contains a description of the proposed approach. 3.1.
Let us specify the entire region of the studied space, in which structural objects of the same nature are obtained, with the coordinates of its starting point and three vectors representing the sides of the parallelepiped. We denote the studied region of space and formally set it as follows:
= 〈 ( 1, 2, 3), 1 1,1, 1,2, 1,3 , 1 2,1, 2,2, 2,3 , 1 3,1, 3,2, 3,3 〉, where ( 1, 2, 3
) stands for the starting point of the studied area of space with coordinates ( 1, 2, 3); ,1, ,2, ,3 is a vector in space = 1,3.
Thus, the vector defines the area of the studied space in the projected system as follows:
= 1, 2, 3, 1,1, 1,2, 1,3, 2,1, 2,2, 2,3, 3,1, 3,2, 3,3 ,
Since, in addition to the general area to which the means of obtaining images are directed, there are sub-areas in which subsets of structural objects are concentrated directly, then such sub-areas are also determined by sets of coordinates within the general area. The definition of the initial coordinates and the coordinates of the vectors of the subdomains are specified by a linear matrix of vectors, which a rectangular matrix can also specify. Division of the area is necessary because subsets of structural objects do not intersect; therefore, they can be examined separately. In addition, it allows scaling performance. Also, in each subset of structural objects, there is their proximity and positioning. Thus, let us define a linear matrix of vectors of subsets of structural objects in the considered region of space as follows: where is a linear matrix of vectors of a subset of structural objects; – vector of i-subset of structural objects, = 1,
; is the number of subsets in the considered region of space.
When detailing the vector values, we set the vector coordinate matrix of a subset of structural objects in the considered area of space as follows: (1) (2) (3) (4) are 12 coordinates of the first vector v_1 according to formula (2), similarly for
= 1, 2, … , , = 1, 1
⋮ 1, 3,3 ⋯ ⋱ ⋯ , 1
⋮ , 3,3 where 1, 1
– 1, 3,3 the remaining
-1 vectors.
The set of coordinates from formula (4) enters the input of the designed system.
The projected system is distributed, as it requires collecting information in a specific space area. acquiring images with the detected objects.
Structural components of an automated system can be in different states. Based on the state matrices reflecting the states of the components of the automated system, a subsystem has been developed for active monitoring of system events and coordinated interaction of the system components when making decisions. This approach enables the system operator to intervene in correcting the behavior of individual components. The automated system should have the following functional capabilities: 1) formation of a UAV group; 2) form a three-dimensional program space and set allowed and forbidden segments; 3) set and adjust the initial flyover points; 4) start a flyby in the working environment of the area of the studied space V; 5) management of output data from the UAV group; 6) change the status of the group and individual UAVs.
In order to activate the operation of the automated system, which provides for the operator to start work and exchange information between software modules, it is necessary to form a group of UAVs that can fly over working segments. Unification of certain features and characteristics of UAVs is highlighted: 1) unique identifier; 2) camera girth width; 3) battery capacity; 4) time spent in flight. These unique characteristics provide registering devices in a system. Considering the above-noted unique features of each UAV, flight parameters that are the same or close to each other are evaluated to form a group. The location of the UAV in the group during the flight depends on the width of the UAV camera coverage. The wider the camera range, the lower the position of the UAV. Any newly created UAV groups in the system have a fixed number of devices that the operator can register. In this work, we consider a group of no more than 4 UAVs for the given task. Taking into account such a characteristic as battery capacity, the group is formed from the approximate technical means of each UAV. During operational time, UAVs can have various types of mechanical damage; therefore, considering the devices ’serviceability, the group can be formed only with working devices.
The next stage in managing a group of UAVs is creating a program mission of overflight of working areas according to the linear matrix (3) of a subset of structural objects. For example, it can be rows of fruit trees.. To consider the main steps of mission management in an automated system, it is assumed that the UAV group does not perform any flight and that the installation of all necessary modules and interactions of all components is completed successfully. Then, the further execution processes taking place during the flyover of the working environment and the functioning of the system are presented in the following steps.
Step 1. Selection and connection of the UAV group to the mission.
1.1. Registered and configured devices in the system are set to the starting points of the flight in the working environment. This step is performed during the startup stages. The operator provides a direct check for mechanical damage and the ability to work with the network. If the test results are determined to be positive, in this case, the operator creates a group of UAVs.
1.2. Camera settings and video quality are checked.
1.3. The state and quality of the network are determined. Special attention is paid to reconnecting to all devices ’networks in case of communication loss.
1.4. During the verification and given the positive results obtained in steps 1.1-1.3 described above, the operator is allowed to create a software mission to fly over the working environment.
Step 2. Management of the overflight mission of the UAV group. The automated system provides control in two modes: 1) initial; 2) automatic.
2.1. In the initial mode, the operator develops the flight path of the working environment by himself. The main goal of this approach is to establish the starting points of work for the entire group of UAVs. The obtained results are analyzed and adjusted to obtain the most accurate data, which is used as input data for the automatic mode. All states in the initial mode of software modules are deterministic, that is, those whose values are set during their initialization. To obtain an entirely predictable result, the operator can enter initial data about the number of structural objects in subsets, for example, fruits on trees, in the working environment for further analysis.
2.2. Making a decision about the further operation of the entire system as a whole according to the data of the initial step is performed thanks to the self-learning system automatically. The automated system allows for adjusting the states of management subsystems. Transitions of the system from one state to another are carried out utilizing events to which the system reacts because of the operation of the UAV group. At this step, the operator can monitor the group’s actions thanks to the monitoring system. At the same time, it is mandatory to use the data of at least one completed whole route to start the automatic control mode of the UAV group.
Step 3. Completion of the UAV group mission.
3.1 Given that the operating environment is challenging to fulfill the operational objectives of the UAV, the operator can complete the mission at any time. The automated system can be transferred to the state of completion, even at the stage of performing its work. It allows sending a group of UAVs to the initial or final point with a defined calculation of the shortest distance to avoid mechanical damage that can be received under the negative influence of weather conditions.
3.2 In case of failure of at least one device from the group, or loss of communication with the network or the system as a whole, if the self-healing subsystem sends a signal to the system about the “critical” state of the system, the mission is recorded as completed, and the software modules receive their proper states, in order to conduct a proper analysis of events in the future.
3.3 In case of loss of communication of the entire group, or at least one UAV in the group, the coordinates of the three-dimensional space of the working environment with the vector number of the subset, such as the number of the fruit row and the place when the device was last detected, are provided to the operator in the monitoring software module.
Providing such information about its geographic location allows the operator to quickly locate the device in the work environment and conduct a proper inspection. The events of a part of the control software modules and the route planning system being in the same state for a long time gives grounds for the fact that a critical mistake has occurred in the operation of the system as a whole, because of which the working mission ends automatically and transfer the group of UAVs to the initial or final points respectively. Proper functioning of the automated system and all its subsystems and appropriate weather conditions in the working environment ensures a “positive” completion of the program mission of the UAV group. 3.2.
The integrated architecture of the developed system is presented by a generalized diagram of the main components in Fig. 2.
In order to evaluate the quality of the planning of the flight of the UAV group, the following criteria for the quality of planning were used: 1) completion of the task in the shortest time; 2) distance during movement in the working environment; 3) resources of hardware devices; 4) the amount of data that the UAV can process during real-time operation. The number of UAVs, flight range, minimum turning radius, and speed range are defined as inputs for the automated system. The distribution of targets among several UAVs, the sequence in which each UAV passes a subset of targets, and the way of moving to each work area is defined as characteristic properties in route planning tasks.
The problem of planning a group of UAVs ’movements is considered a mathematical problem for a traveling salesman. A schematic representation of the method of controlling several UAVs in an automated system is shown in Fig. 3.
Since the information system consists of many hardware devices and software modules that interact through the network, all work is managed with the help of operators. The following characteristic properties of management are distinguished: 1) setting, support, and monitoring of the real-time kinematics network; 2) forming a group of UAVs according to their characteristic properties; 3) performance of tasks for monitoring the integrity of the information system; 4) creation of a software coordinate working environment; 5) formation of the initial program coordinates of the flyby. The operator solves tasks at all stages of the information system’s life cycle, but thanks to the developed self-organized and decentralized software modules, his involvement in fulfilling the purpose of the task is minimal.
Within the framework of the automated system (Fig. 1), it is provided for setting destination points (in other words, points in the considered environment), which each drone must reach within its segment and mark on the generated map. At the same time, each point is a set of coordinates in space. A set of such points forms a route that the UAV must fly once.
According to the conducted experiments in a natural environment, it was established that to calculate the number of structural objects of a subset (for example, fruits from one tree), it is advisable to combine several UAVs into one group. As a result, a certain number of UAVs with their coordinates in space was obtained. The initial coordinates for the overflight of one row of the working environment were plotted on the terrain map, which is provided by a coordinate grid. As a result, a specific matrix of states is obtained, which displays physical devices with their coordinates and points of completed tasks.
This representation of components through the states that hardware devices can be in during operation allows for determining the state of operation and monitoring critical situations or failures. Such characteristics made it possible to form a set of coordinates for constructing a quasi-optimal route. Analysis of information from the matrix of states enables the automated system to make decisions about further actions of each device. A centralized subsystem has been developed to manage a group of UAVs. An essential element of the module is its ability to be open for the integration of other subsystems. The research object is determined by many dynamic external factors, namely, different weather conditions and limitations of UAV power and throughput resources. Such factors have a negative impact on the quality of recognition, and as a result, on the correctness of calculating their number. Moreover, this, in turn, reduces the time of work, reduces the distance that the UAV can fly, reduces computing capabilities, and limits communications with the center of the subsystem. At the same time, in order to avoid duplication and recognition of extraneous objects, UAVs must fly over economically efficient routes. However, the route planner must use efficient path-planning approaches that minimize total flight length.
¯ , = 1, , is calculated as follows:
So, to achieve the research goal and consider the above limitations, a route planning method with self-learning technology was developed. This method was developed based on the algorithmic QLearning (QL) approach of artificial intelligence Reinforcement Learning (RL) [22 - 24]. The route planning method for managing a group of UAVs is implemented as a centralized subsystem. The system module that implements the route planning method generates a state matrix with UAV group coordinates, which, according to the QL algorithm, all devices in three-dimensional space can adjust their behavior through interaction with the working environment. The subsystem forms state matrices for each UAV and works according to the “action-reward” approach, which a so-called intelligent agent performs. All hardware devices in the group perform their work until they reach “positive completion” of the automated system [25, 26]. An essential element of the module is its ability to be open for integration with other subsystems.
The route planning module performs an iterative process of automatically adjusting the UAV route. Due to the iterative actions of each UAV, the module generates and continuously updates the Q-value for each drone until the most accurate value is reached, which can be considered quasioptimal. Q-value for each action of each UAV and for each of its states at the moment of time ( , ) = ( −1 , −1 ) + −1 + ∙ { ( , )} − ( −1 , −1 ) ,
(5) where ( −1 , −1 ) is a quantitative expression of the reward received by the intelligent agent for reaching the previous state −1 ; is the learning speed coefficient of the model, 0 < ≤ is the level of reward received by the intelligent agent in case of transition from state −1 to state ; is the depreciation coefficient, which determines the importance of future rewards obtained by 1; an intelligent agent, 0 ≤ ≤ 1;
{ ( , )} is the estimated quantitative value of the future reward in case of performing action
while in state .
An example of the result of the route planning module is given in Table 1.
The developed subsystem for implementing the route planning method (Fig. 4) forms quasioptimal routes through continuous learning, interacting with the working environment through an iterative process. The main components of the trajectory planning subsystem, which are obtained automatically, are highlighted in Fig. 5.
By “Segment”, we present a set of UAVs connected by software characteristics that fly over the specified working environment segments and visit the maximum number of working zones while using the shortest path with minimum delay. The main characteristics of the experimental environment are the division into permitted zones and prohibited working segments. Here, “State” is a software structural part that performs self-monitoring regarding analyzing the location of a set of UAVs in coordinate space. When staying in the same state for a long time, processing and analysis of current tasks are carried out in automatic mode and, if necessary, transferred to another state of the entire group of devices. Given the short duration of the UAV operation, an additional module is also integrated to manage the state of the UAV battery resources. Such an implementation allows managing all possible states, both software, and hardware.
The primary states of route planning are described below. • Status “Action.” The transition from one state to another occurs due to a change in specific parameters that characterize the automatic route generation subsystem. The leading state of such a structural software subsystem is an action that a set of devices performs by flying to target points thanks to the formed trajectories in the coordinate space. • State “Environment.” Representation of the three-dimensional coordinate space is the basis for building mechanisms and determining the characteristics of automatic trajectories. The experimental environment is divided into permitted and prohibited segments thanks to the software module. • Status “Reward.” The resulting values of the “smart” route planning module. The component generates a response in the form of states thanks to an iterative approach and defining characteristics of the working environment. The centralized system evaluates module states thanks to a series of actions. • State “Completion.” It determines the formation of the completion of the mission of the UAV group. Structural components that track the states of all other subsystems form the termination mechanism. Based on specific parameters of the state of the entire mission, the component decides to terminate. If all work segments are passed and all assigned tasks are completed, the system enters the “positive completion” state. The life cycle of the mission can be stopped if the module receives parameters according to which a group of hardware devices should go into the “standby” state.
Overall, the developed architecture of the self-learning subsystem is distributed and multi-level. This presentation of components through the states in which the software modules can be during operation determines the integrity state of the entire software system and the integrity states of its components. Our contribution allows for increasing the number of levels of subsystems without changing its architecture.
Since, after installation, the system works in the “initial” mode, the input data for the analysis and comparison of the target operation of the UAV is the calculated number of structural objects in given subsets of the studied area of space, for example, the number of fruits on individual trees in one row of the working environment. The data was obtained in a certain number of “positive” completed missions to compare the indicators of similarity between the current data obtained in the automatic and initial modes.
The results of experimental studies are presented in Table 2. The results from Table 2 confirm the possibility of implementing the proposed solutions.
Receiving answers about the states and results during the work of the program mission is submitted to the monitoring module. By analyzing this data, the operator can guide future actions and adjust the states of subsequent missions to improve the performance evaluation criteria values of the UAV group. Processing by the software module of uncertainties associated with the lack of appropriate states of various subsystems leads to the formation of a report on errors in the system. The operator can use the received data to adjust the system’s appropriate parts and individual hardware parts in the working environment.
One of the main factors in the success of the program mission of the UAV group is communication with the network. To build a system that in real time calculates the number of structural objects, for example, fruits on trees, it is essential to receive information about the state of the network and to be able to control devices that are installed in the working environment, for example, in an orchard. The monitoring software module monitors the state of the network and reports all critical states of communication with the group and each device separately.
Thus, the automated system allows organizing the UAV overflight of the studied space area subsets according to the given initial data. Its architecture allows for achieving the appropriate level of organization when determining the next steps in functioning subsystems and components.
The developed architecture of the automated system for the dynamic acquisition of images of structural objects in three-dimensional space is the basis for creating new tools that can fly UAVs over subsets of the studied area of space according to the given initial data. It allows for achieving the appropriate level of organization when determining the next steps in functioning subsystems and components. The proposed self-organized automated system is aimed at solving the critical problems, namely: 1) reducing the delay time during the execution of the task; 2) reducing the distance when moving in the working environment; 3) optimization of hardware device resources; 4) increasing the amount of data that the UAV can process during real-time operation.
Management tools perform software mission management by combining a fixed number of UAVs into a group and performing targeted work in fragments of the working environment. The automated system’s monitoring software module processes the mission’s initial data and conducts analyses and comparisons based on already valid data to ensure the most accurate result of calculations of the number of fruits on trees.
Further research directions are improving the flight methods implemented in it, image recognition, and calculation of the number of recognized objects.
operation support for fleets of drones, in 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Mar. 2019, pp. 931–936. doi:10.1109/PERCOMW.2019.8730838 [12] E. Corcoran, M. Winsen, A. Sudholz, and G. Hamilton, Automated detection of wildlife using drones: Synthesis, opportunities and constraints, Methods in Ecology and Evolution, vol. 12, no. 6, pp. 1103–1114, 2021, doi:10.1111/2041-210X.13581 [13] W. Yu, T. Yang, and C. Chen, Towards resolving the challenge of long-tail distribution in UAV images for object detection, presented at the Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 3258–3267. Accessed: Feb. 11, 2023. [Online].
Available: https://bit.ly/3lu9UNV [14] A. Ivanov, B. Muhammad, K. Tonchev, A. Mihovska, and V. Poulkov, UAV-based volumetric measurements toward radio environment map construction and analysis, Sensors, vol. 22, no. 24, Art. no. 24, Jan. 2022, doi:10.3390/s22249705 [15] S. Alexandris et al., Integrating drone technology into an innovative agrometeorological methodology for the precise and real-time estimation of crop water requirements, Hydrology, vol. 8, no. 3, Art. no. 3, Sep. 2021, doi:10.3390/hydrology8030131 [16] Y. Taddia, L. González-García, E. Zambello, and A. Pellegrinelli, Quality assessment of photogrammetric models for façade and building reconstruction using DJI Phantom 4 RTK, Remote Sensing, vol. 12, no. 19, Art. no. 19, Jan. 2020, doi:10.3390/rs12193144 [17] T. Gao and X. Bai, Bayesian optimization-based three-dimensional, time-varying environment monitoring using an UAV, J Intell Robot Syst, vol. 105, no. 4, p. 91, Aug. 2022, doi:10.1007/s10846-022-01709-x [18] D. Xiong et al., UAV Inspection Monitoring and Management System based on Wireless Sensing Technology, in 2022 IEEE 2nd International Conference on Electronic Technology, Communication and Information (ICETCI), May 2022, pp. 958–961. doi:10.1109/ICETCI55101.2022.9832098 [19] T. Bernstein and V. Janssen, Positional uncertainty of network RTK observations in a modern datum, Journal of Geodetic Science, vol. 11, no. 1, pp. 38–47, Jan. 2021, doi:10.1515/jogs-20200116 [20] D. Hu, V. J. L. Gan, T. Wang, and L. Ma, Multi-agent robotic system (MARS) for UAV-UGV path planning and automatic sensory data collection in cluttered environments, Building and Environment, vol. 221, p. 109349, Aug. 2022, doi:10.1016/j.buildenv.2022.109349 [21] S. Alshattnawi, L. Afifi, A. M. Shatnawi, and M. M. Barhoush, “Utilizing Genetic Algorithm and Artificial Bee Colony Algorithm to Extend the WSN Lifetime”, IJC, vol. 21, no. 1, pp. 25-31, Mar. 2022, doi.org/10.47839/ijc.21.1.2514 [22] P. Radiuk and N. Hrypynska, A framework for exploring and modelling neural architecture search methods, in Proceedings of the 4th International Conference on Computational Linguistics and Intelligent Systems (CoLInS-2020), Lviv, Ukraine, April 23-24, 2020, 2020, vol. 2604, pp. 1060–1074. Accessed: May 30, 2020. [Online]. Available: http://ceur-ws.org/Vol-2604/ [23] Y. Alon and H. Zhou, Multi-agent reinforcement learning for unmanned aerial vehicle coordination by multi-critic policy gradient optimization. arXiv, Dec. 31, 2020. doi:10.48550/arXiv.2012.15472 [24] O. Savenko, S. Lysenko, and A. Kryschuk. Multi-agent based approach of botnet detection in computer systems. Communications in Computer and Information Science 291 (2012) 171-180 doi: 0.1007/978-3-642-31217-5_19 [25] V. Yatskiv, N. Yatskiv, Su Jun, A. Sachenko and Hu Zhengbing, "The use of modified correction code based on residue number system in WSN," 2013 IEEE 7th International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS), Berlin, 2013, pp. 513-516, doi: 10.1109/IDAACS.2013.6662738 [26] Nemati M, Al Homssi B, Krishnan S, Park J, Loke SW, Choi J. Non-Terrestrial Networks with UAVs: A Projection on Flying Ad-Hoc Networks. Drones. 2022; 6(11):334. https://doi.org/10.3390/drones6110334