Algorithms for obtaining video and sound data of UAVs in real time

Abstract

This paper investigates real-time data acquisition algorithms on unmanned aerial vehicles (UAVs) between the flight controller and additional equipment. The main attention is paid to data exchange algorithms between companion computers (Raspberry Pi type) and microcontrollers (Arduino type) additionally installed on the drone itself. An important aspect of the effective operation of the UAV is the fast and reliable transmission of data, in particular sound, between its components. This provides an accurate simulation of its behavior. The article discusses data exchange algorithms to reduce delays and increase reliability. The work focuses on determining requirements for real-time data exchange for UAV systems, analysis of limitations and requirements for speed, reliability. An overview of the principles of operation of microcontrollers and minicomputers, highlighting their differences and advantages of joint use, is carried out. An analysis of existing data exchange algorithms and protocols (SPI, UART, I2C, etc.) and Schauder's direct discrete transformation algorithm, which is an integral part of the direction of research on coding and decoding of audio information, was also performed, with the aim of comparing their characteristics and capabilities. This research contributes to the improvement of UAV data exchange technologies, offering new approaches and solutions that can be useful for developers and researchers in the field of unmanned technologies.

Keywords

• eol>UAV
• data acquisition algorithm
• microcontroller
• microcomputer
• data exchange protocol
• discrete Schauder transformation 1

1. Introduction

In the modern world, unmanned aerial vehicles (UAVs) have become widely used in various fields, including agriculture, search and rescue operations, infrastructure monitoring, and the main one is their use in the war with the Russian aggressor. One of the key requirements for the effective functioning of UAVs is the possibility of fast and reliable data exchange between various components of the drone in real time. This allows for accurate modeling of its behavior, quick response to changes in the environment. Various types of additional hardware are installed on the drone to simulate its behavior, including flight controllers, companion computers (such as Raspberry Pi), and microcontrollers (such as Arduino). The joint use of these components allows the implementation of complex data processing and decision-making algorithms. However, to achieve maximum speed, reliability and energy efficiency, it is necessary to ensure effective data exchange between these components. In this article, the main attention is paid to data exchange algorithms between companion computers and microcontrollers installed on the drone. The methods of data exchange optimization are considered, which allow to reduce delays and increase the reliability of the system. The paper analyzes in detail the requirements for real-time data exchange for UAV systems, in particular the requirements for the speed and reliability of information transmission. An overview of the principles of operation of microcontrollers and minicomputers, highlighting their differences and advantages of joint use, is carried out.

2. Review of existing solutions and literature sources

In article [ 1 ], data collection from deployed sensor networks can be performed using a static receiver, a ground mobile receiver, or a mobile aerial data collection based on an unmanned aerial vehicle (UAV). Considering the large-scale sensor networks and the characteristics of the deployed environment, aerial data collection based on manned UAVs has more advantages. In this paper, the authors developed a basic framework for aerial data collection, which includes the following five components: network deployment, node positioning, reference point search, UAV fast route planning, and network data collection. In each of them, the authors identified key problems and proposed effective solutions. This includes the proposal of a Fast Route Planning by Algorithm Rules (FPPWR) algorithm based on network distribution to improve the efficiency of route planning while ensuring a relatively short path length. The authors developed and implemented a modeling platform for collecting aerial photo data from sensor networks and tested the performance of the proposed system based on the following parameters: time spent on aerial photo data collection, flight path distance, and the amount of data collected. The disadvantage of this work is that the methods proposed by the authors do not solve the problem of obtaining sound and video data from UAVs.

The authors of the article [ 2 ] noted that the use of unmanned aerial vehicles (UAVs) is considered an effective platform for monitoring critical infrastructure covering geographical areas. UAVs have also demonstrated exceptional viability in data collection due to the extensive wireless sensor networks they operate within. Based on environmental information such as restricted airspace, geographic location conditions, flight risks, and sensor deployment statistics, we design an optimal flight path planning mechanism using biologically inspired multi-objective algorithms. In this paper, the authors first collect data detection points from the entire sensor field in which the UAV communicates with sensors to acquire sensor data, and then determine the best flight path between neighboring collection points. Using the proposed joint genetic algorithm and ant colony optimization from the possible UAV flight routes, the optimal one is selected according to detection utilities, energy, time, and risk. The simulation results show that the method synthesized by them can obtain dynamic adaptability to the environment and high utility in various practical situations. However, the authors of this work also do not pay special attention to the analysis of video and audio information.

In [ 3–7 ], it is noted that unmanned aerial vehicles (UAVs) are increasingly used as data collectors for terrestrial wireless sensor networks (WSN). Most of the current research suggests optimization for creating routes for a single UAV. In contrast, the authors propose a distributed algorithm for WSN data collection using a dynamic UAV array that takes into account that UAVs leave or join the cluster due to reboots or failures. In their work, the authors believe that UAVs only have mediumrange (several meters) communication capability to deliver collected data, similar to assumptions in related papers. Compared to the expensive non-real-time traveling salesman problem (TSP) approach, our algorithm provides about 3% more efficient sensor visits in certain scenarios without using optimized traversal, which is a significant drawback of this study.

The article [8–10] states that artificial data collection from distributed sensors located in different areas in complex scenarios is obviously inefficient due to the large amount of work and time. Unmanned aerial vehicles (UAVs) are a promising solution that allows several UAVs to automatically collect data along a predetermined route. However, without a well-planned trajectory, the required number and energy consumption of unmanned aerial vehicles will increase dramatically. Therefore, minimizing the required number and optimizing the UAV path, known as multi-UAV route planning, is essential for effective data collection. Therefore, some heuristic algorithms such as Genetic Algorithm (GA) and Ant Colony Algorithm (ACA) have been proposed, which work well for multiUAV route planning. However, in complex scenarios with high timeliness requirements, the convergence speed performance of the above algorithms is imperfect, which will lead to inefficient optimization process and data collection delay. Deep learning (DL, DP), after training with enough data sets, has a high resolution speed without worrying about convergence problems.

Therefore, in this paper, the authors propose an algorithm called Deep Learning with Genetic Algorithm (DL-GA), which combines the advantages of DL and GA. The GA will collect states and routes from different scenarios and then use them to train a deep neural network so that when faced with familiar scenarios, it can quickly provide an optimized route that can meet high operational demands. Numerous experiments show that the solving speed of DL-GA is much higher than that of GA, with almost no loss of optimization ability, and can even outperform GA under certain conditions.

The work [8, 9] states that due to the advantages of deployment flexibility and high mobility, unmanned aerial vehicles (UAVs) have found wide application in the fields of disaster relief, crop protection, environmental monitoring, etc. With the development of unmanned aerial vehicles and sensor technologies, UAV data collection for the Internet of Things (IoT) is attracting increasing attention.

This article examines key UAV data collection scenarios and technologies in detail. First, we present a system model including a network model and a mathematical model of UAV data collection for IoT. The authors review key technologies, including sensor clustering, UAV data collection mode, and joint route planning and resource allocation. Finally, open problems are discussed in terms of efficient multiple access and collaborative discovery and data collection. This paper provides some recommendations and ideas for researchers in the field of UAV data collection for IoT.

In this article, special attention is paid to the analysis of existing algorithms and data exchange protocols, such as SPI, UART, I2C [11–13] and Schauder's direct discrete transformation algorithm [14, 15], in order to study their characteristics and capabilities. This research can contribute to the improvement of UAV data exchange technologies, offering new approaches and solutions that can be useful for developers and researchers in the field of unmanned technologies.

3. Methodology for the study algorithms for obtaining video and sound data of UAV

Some of the popular and affordable flight controllers today are SpeedyBee and Matek with Betaflight or Ardupilot firmware. In this study, it is proposed to take as an example one of the most common options - SpeedyBee with Betaflight firmware.

Before going into detail, the physical interfaces that are present on these flight controllers will be considered. Both SpeedyBee and Matek flight controllers with Betaflight firmware and Ardupilot flight controllers include UART (Serial), I2C and SPI. These interfaces allow you to connect various devices such as GPS, OSD, additional sensors and other devices that extend the capabilities of the drone and of course Raspberry and Arduino.

Consider the available interfaces on Raspberry Pi and Arduino: 1. Raspberry Pi: •

GPIO (General Purpose Input/Output): This is the main interface for connecting various devices and sensors to the Raspberry Pi. It allows you to read input signals from sensors and control output signals to actuators or other devices. • • • • • • • • 2. Arduino:

Digital Pins: These digital pins can be used as input or output to read or set logic levels. Analog Pins: Arduino also has analog pins for reading analog signals from sensors or other sources.

UART (Serial): This interface allows you to send and receive data via a serial connection to other devices, such as a Raspberry Pi or a flight controller.

SPI (Serial Peripheral Interface): Arduino also supports SPI interface to interface with additional devices such as displays, SD cards, sensors, etc.

I2C (Inter-Integrated Circuit): This interface allows the Arduino to connect to additional sensors, displays, and other devices using the I2C bus.

The general idea is that we use the Raspberry Pi as a photo and video image processing and behavior simulation tool, and the Arduino as an intermediate bridge to exchange data between the flight controller and the Raspberry Pi. For example, Arduino is used to transmit and generate control signals, barometer data and other sensors to Raspberry Pi and flight controller (Figure 1).

By analyzing the available data exchange protocols and the collected information, we can propose an exchange algorithm in which data is transferred from the flight controller to the Arduino and then to the Raspberry Pi via existing protocols. The goal of this is to create a simple framework that allows developers to easily test their hypotheses for modeling the behavior of an unmanned aerial vehicle, lowering the barrier to entry and allowing them to focus on algorithm development rather than data communication system development.

Below is a Table 1 for comparing the characteristics of these protocols.

UART (Universal Asynchronous Receiver/Transmitter): This interface allows you to send and receive data over a serial connection to other devices, such as an Arduino or a flight controller.

SPI (Serial Peripheral Interface): The SPI interface is used to interface the Raspberry Pi with additional devices such as sensors, displays, and other peripherals.

I2C (Inter-Integrated Circuit): This interface allows the Raspberry Pi to connect to additional sensors, displays, and other devices with multiple devices on a single bus line.

Up to 10 Mbps and more 4 (MOSI, MISO, SCK, 2 (SDA, SCL) SS) May work with several Up to 127 devices devices (limited number SS lines) Synchronous Synchronous high low

In conclusion, we can say that all the necessary devices have the same interfaces for exchanging information and can be used to solve the tasks and how they can be used to connect to other components of the UAV system.

It is proposed to use the discrete Schauder transformation to exchange sound information, for designing and researching network means of encoding / decoding, which can be implemented at the algorithmic-program level or in a hardware-technological design based on an integrated circuit of the type. The mathematical basis of such studies is based on the previous results of the publications of the authors of this work, for example [16].

From a positive point of view, the use of the Schauder transformation has certain advantages compared to trigonometric bases, for example, the possibility of local processing on the time and frequency interval when segmenting the incoming sound stream, reducing the time for mathematical calculations of the expansion coefficients, minimizing the amount of memory. All this significantly affects the speed of data delivery to the user.

The image presented in Figure 2 is chosen for research [17–20].

Within the framework of this work, we present the direct discrete Schauder transformation algorithm, which is an integral part of the direction of research on the subject of coding and decoding of audio information (Figure 3).

In Figure 3, the indices k, m, n determine the serial numbers of digitized code combinations of segmented blocks 1,2,3... L of the sound file in accordance with the coordinates of the Schauder functions on the interval [0, T]. The relationship between single and double numbering of functions is as follows [16]: 1,2,3, … , ; 1,2,3, … , 2 ;

T = 2MΔt, where Δt is the quantization interval of the sound signal over time. 2 ,

A structural-functional model of an experimental setup for further, more detailed research is presented in Figure 4 for the purpose of comparison with other standardized formats of encoding/decoding audio information, i.e., software implementation of audio codec functions at a representative level, using the discrete Schauder transformation [21–23].

Further studies of the practical use of Schauder's non-orthogonal basis functions, taking into account the work of the authors of this publication, can be focused on the development of effective algorithmic software and hardware for multimedia representation and compression of audio and video streams in flight information processing systems, moving (mobile) objects, for example, UAVs [24–28].

4. Experimental studies

The above data exchange protocols support significant data exchange rates on all three devices (Arduino, flight controller, Raspberry Pi), for example: • •

UART - speed is measured in baud, which is the number of symbols transmitted per second. If there are exactly two symbols in the system (usually 0 and 1), then baud and bits per second (bps) are equivalent.

SPI - speed is measured in Mbit/s.

Depending on the device, speeds can be more than 500,000 baud (bit/s) or 1 Mbps. Let's imagine that we need to transmit from the Raspberry Pi to the flight controller through the Arduino the control signals of the motors or other information that can be expressed in a numerical value, for example, from 0 to 2000 or more, for example, 10 different values, at a speed of 115200 baud (bit/s). So what speeds are we talking about - let's make a general calculation without taking into account the specifics of data packet formation, which additionally uses several bytes of information or the time for packet formation, which creates a delay of several percent of the selected speed: One number from 0 to 2000 can be represented as two bytes (16 bits), that is, 10 numbers are 20 bytes or 160 bits. Time for transmission of one packet of information [29–33]: where: t - transmission time; Nbit - number of bits S - transmission rate (bit/s).

For this case: • •

Nbit = 160;

S = 115,200 bps.

Then the transmission time will be: 160 біт

0.0013888 s 115 200 біт/с Therefore, the transmission frequency (number of packets per second) is defined as: 1 ! .

For this case:

! "."" #$$$ % 740 .

Thus, it is possible to transmit information at a frequency of approximately 720 Hz, which is much higher than 60 frames per second (Hz) when compared to the operating speed of, for example, conventional video cameras, which is likely to be sufficient for modeling the behavior and control of UAVs through image processing and making decisions according to a predetermined logic. Taking into account the information above, it becomes quite clear that information transfer rates allow for very high speeds to organize information exchange and transfer data between devices. This will most likely be sufficient for the functioning of the system in accordance with the constructed algorithm [34–39].

Using Raspberry Pi for video processing. Assuming that the Raspberry Pi has enough image processing potential, I suggest using this device to analyze video and photo data coming from an unmanned aerial vehicle (UAV). When considering the image processing capabilities of the Raspberry Pi, it can be used, for example, to orient along an optical channel or to make decisions based on video or photo data. This opens up wide opportunities for expanding the functionality of the system and improving its response to the environment.

Information exchange algorithm. It is important to remember to solve the following issues in the process of data exchange at high speeds:

Check data integrity, for example, by signing a data packet with a CRC-8 checksum (C++ code example): uint8_t crc8(const uint8_t *data, size_t len) { uint8_t crc = 0x00; for (size_t i = 0; i < len; i++) { crc ^= data[i]; for (uint8_t j = 0; j < 8; j++) {

crc = crc & 0x80 ? (crc << 1) ^ 0x31 : crc << 1; } } return crc; • •

Ensure synchronicity/sequence of receiving and exchanging data between devices, for example, by organizing function calls by time.

5. Discussion of research results

As a result of the study, it was established that the UART, SPI and I2C information exchange protocols can be effectively used for data transfer between the UAV flight controller, Arduino-type microcontrollers and Raspberry Pi-type minicomputers. This is confirmed by the recommendations of the flight controller developers themselves. The controller modified for this study is shown in Figure 5.

The study also showed that it is possible to simulate the behavior of a drone on a Raspberry Pi computer and transmit control commands to the UAV. Information exchange protocols are able to work at sufficiently high speeds, which allows to ensure the necessary speed and reliability of data transmission. However, in order to achieve maximum efficiency, it is necessary to conduct further research on the optimization of the algorithm for the use of these protocols. Figure 6 shows the UAV flight simulation process at the current stage of development.

In further research, it is planned to improve the algorithm and information exchange framework between the flight controller, Raspberry Pi and Arduino at the software level.

6. Conclusions

The proposed algorithms and approaches to data exchange between UAV components are relevant and important to ensure their effective operation. The use of microcontrollers and minicomputers makes it possible to implement complex data processing algorithms, which is necessary for modern unmanned systems.

Analysis of quantitative parameters shows that SPI has the highest data transfer rate (up to 10 Mbit/s), but requires more communication lines. I2C provides support for up to 127 devices, which makes it attractive for complex systems with a large number of components, but it has a relatively complex implementation. UART is the simplest to implement with low hardware requirements, but supports only one device per communication line.

In terms of quality, UART stands out for its simplicity of implementation and low power consumption, which makes it attractive for resource-constrained systems. SPI, although high in power consumption, provides high data rates and supports multiple devices, which can be useful in systems that require fast, multi-channel data transfer. I2C, although more complex to implement, provides low power consumption and the ability to connect a large number of devices, making it the optimal choice for complex systems with many components.

The analysis of quantitative parameters confirms that the proposed methods of data exchange can provide high speeds of information transmission between UAV components, which is important for the real-time functioning of these systems. The proposed approaches make it possible to achieve a data transfer rate of up to 720 Hz, which significantly exceeds the minimum frequency required for video cameras (60 Hz).

The proposed approaches for using the discrete Schauder transform for audio information exchange can significantly improve data processing speed and reduce latency. This opens up new opportunities for the development of effective algorithmic and software solutions for multimedia presentation and compression of audio and video streams in information processing systems of moving objects, such as UAVs.

Overall, the study contributes to the improvement of data exchange technologies in unmanned systems, suggesting new approaches and solutions that can be useful for developers and researchers in the field.

Declaration on Generative AI

The author(s) have not employed any Generative AI tools. [26] V. Kalashnikov, S. Dempe, B. Mordukhovich, S.V. Kavun, Bilevel optimal control, equilibrium, and combinatorial problems with applications to engineering, Mathematical Problems in Engineering 2017 (2017) 7190763. [27] A. Dudnik, O. Trush, V. Kvasnikov, T. Domkiv, Development of distributed multi-segment wireless networks for determining external situations, CEUR Workshop Proceedings 2845 (2021) 127–137. URL: https://ceur-ws.org/Vol-2845/Paper_13.pdf. [28] O. Pysarchuk, A. Gizun, A. Dudnik, V. Griga, T. Domkiv, S. Gnatyuk, Bifurcation Prediction Method for the Emergence and Development Dynamics of Information Conflicts in Cybernetic Space, CEUR Workshop Proceedings 2654 (2019) 692–709. URL: https://ceur-ws.org/Vol2654/paper54.pdf. [29] A. Dudnik, Y. Kravchenko, O. Trush, O. Leshchenko, N. Dakhno, V. Rakytskyi, Study of the Features of Ensuring Quality Indicators in Multiservice Networks of the Wi-Fi Standard, in: Proceedings of 3rd International Conference on Advanced Trends in Information Theory (ATIT), IEEE, Kyiv, Ukraine, 2021, pp. 93–98. doi: 10.1109/ATIT54053.2021.9678691. [30] L. Kuzmych, D. Ornatskyi, V. Kvasnikov, A. Kuzmych, A. Dudnik and S. Kuzmych, "Development of the Intelligent Instrument System for Measurement Parameters of the Stress Strain State of Complex Structures, in: Proceedings of 4th International Conference on Advanced Trends in Information Theory (ATIT), IEEE, Kyiv, Ukraine, 2022, pp. 120–124. doi: 10.1109/ATIT58178.2022.10024222. [31] I. Bakhov, Y. Rudenko, A. Dudnik, N. Dehtiarova, S. Petrenko, Problems of Teaching Future Teachers of Humanities the Basics of Fuzzy Logic and Ways to Overcome Them. International Journal of Early Childhood Special Education (INT-JECSE) 13(2) (2021) 844–854. doi: 10.9756/INT-JECSE/V13I2.211127. [32] O. Trush, A. Dudnik, M. Trush, O. Leshchenko, K. Shmat and R. Mykolaichuk, "Mask Mode Monitoring Systems Using IT Technologies," 2022 IEEE 4th International Conference on Advanced Trends in Information Theory (ATIT), Kyiv, Ukraine, 2022, pp. 219–224, doi: 10.1109/ATIT58178.2022.10024216. [33] A. Dudnik, B. Presnall, M. Tyshchenko, O. Trush, Methods of determining the influence of physical obstructions on the parameters of the signal of wireless networks, CEUR Workshop Proceedings 3197 (2021) 227–240. URL: https://ceur-ws.org/Vol-3179/Paper_21.pdf. [34] N. Dakhno, O. Leshchenko, Y. Kravchenko, A. Dudnik, O. Trush, V. Khankishiev, Dynamic Model of the Spread of Viruses in a Computer Network Using Differential Equations, in: Proceedings of 3rd International Conference on Advanced Trends in Information Theory (ATIT), IEEE, Kyiv, Ukraine, 2021, pp. 111–115. doi: 10.1109/ATIT54053.2021.9678822. [35] R. Aleksieieva, A. Fesenko, A. Dudnik, Y. Zhanerke, Software tool for ensuring data integrity and confidentiality through the use of cryptographic mechanisms, CEUR Workshop Proceedings 3426 (2023) 259–273. URL: https://ceur-ws.org/Vol-3426/paper21.pdf. [36] A. Dudnik, S. Dorozhynskyi, S. Grinenko, O. Usachenko, B. Vorovych, O. Grinenko, Methods of constructing a lighting control system for wireless sensor network “Smart Home”, Advances in Artificial Systems for Logistics Engineering Lecture Notes on Data Engineering and Communications Technologies (2022) 170–179. doi: 10.1007/978-3-031-04809-8_15. [37] A. Dudnik, I. Bakhov, O. Makhovych, Y. Ryabokin, O. Usachenko, Models and methods for improving performance of wireless computer networks based on the decomposition of lower layers of the OSI reference model, International Journal of Emerging Technology and Advanced Engineering 12 (2022) 152–162. doi: 10.46338/ijetae0122_15. [38] A. Bieliatynskyi, S. Yang, V. Pershakov, M. Shao, M. Ta, The use of fiber made from fly ash from power plants in China in road and airfield construction, Construction and Building Materials 323 (2022) 126537. doi: 10.1016/j.conbuildmat.2022.126537. [39] O. Trush, I. Kravchenko, M. Trush, O. Pliushch, A. Dudnik, K. Shmat, Model of the sensor network based on Unmanned Aerial Vehicle, in: Proceedings of 3rd International Conference on Advanced Trends in Information Theory (ATIT), IEEE, Kyiv, Ukraine, 2021, pp. 138–143. doi: 10.1109/ATIT54053.2021.9678623.

References

[1] C. Wang , F. Ma , J. Yan , D. De, S. K. Das , Efficient aerial data collection with UAV in large-scale wireless sensor networks , International Journal of Distributed Sensor Networks 11 ( 11 ) ( 2015 ). doi: 10 .1155/ 2015 /286080.

[2] Q. Yang , S. -J. Yoo , Optimal UAV path planning: Sensing data acquisition over iot sensor networks using multi-objective bio-inspired algorithms , IEEE Access 6 ( 2018 ) 13671 - 13684 . doi: 10 .1109/ACCESS. 2018 . 2812896 .

[3] B. Olivieri , M. Endler , An algorithm for aerial data collection from wireless sensors networks by groups of UAVs , in: Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Vancouver, BC, Canada, 2017 , pp. 967 - 972 , doi: 10.1109/IROS. 2017 . 8202262 .

[4] Y. Pan , Y. Yang , W. Li , A deep learning trained by genetic algorithm to improve the efficiency of path planning for data collection with multi-UAV, IEEE Access 9 ( 2021 ) 7994 - 8005 . doi: 10 .1109/ACCESS. 2021 . 3049892 .

[5] R. S. Odarchenko , S. O. Gnatyuk , T. O. Zhmurko , O. P. Tkalich , Improved method of routing in UAV network , in: Proceedings of International Conference Actual Problems of Unmanned Aerial Vehicles Developments (APUAVD) , IEEE, Kyiv, Ukraine, 2015 , pp. 294 - 297 . doi: 10 .1109/APUAVD. 2015 . 7346624 .

[6] J. Gong , T.-H. Chang , C. Shen , X. Chen , Flight Time Minimization of UAV for Data Collection Over Wireless Sensor Networks , IEEE Journal on Selected Areas in Communications 36(9) ( 2018 ) 1942 - 1954 . doi: 10 .1109/JSAC. 2018 . 2864420 .

[7] C. You , R. Zhang, 3D trajectory optimization in rician fading for UAV-enabled data harvesting , IEEE Transactions on Wireless Communications 18 ( 6 ) ( 2019 ) 3192 - 3207 . doi: 10 .1109/TWC. 2019 . 2911939 .