In this paper , we present a simulator that replicates a communication channel between terrestrial users using low-altitude airborne mobile stations (Unmanned Aerial Vehicles). Through a Q-Learning network, the UAVs determine the optimal position to occupy to improve transmission between users and converge towards these optimal states. This system lends itself to various applications, including the restoration of the telephone network in areas without coverage or afected by natural disasters, and the establishment of mobile radio bridges to connect remotely controlled vehicles in conflict scenarios, making communications dificult to intercept. The results presented were obtained by tailoring the model in diferent scenarios, according to a realistic concentration and distribution of buildings in the four analyzed environments
parameters of wireless communication, including (i) connection throughput and (ii) number of failed connections.
Recent developments in international conflicts have It is worth noting that there are already studies
focusshown how the use of drones has influenced the course ing on modeling a telecommunications system with
lowof the most significant war scenarios. Even in this con- altitude mobile repeaters. These systems aim to find the
text, machine learning is finding increasing applications, optimal altitude of the aircraft given a fixed transmission
enhancing the capabilities and precision of drone opera- power [
This article proposes an application of UAVs in the coverage area [
In accordance with recent studies, future
communicatio networks are anticipated to integrate non-terrestrial 2.1. Parameters and their description
networks, including Low Earth Orbit (LEO) satellites and The descriptive parameters of the simulated model are
High Altitude Platform Systems (HAPS) utilizing Un- divided into the following three macro groups:
manned Aerial Vehicles (UAVs) [
Among the studies present in the literature, various systems for managing the movement of UAVs have been experimented with: centralized and distributed systems.
The centralized system consists of a terrestrial radio
station for estimating channel parameters and
controlling the telemetry signals of the UAVs [
It consists of a set of parameters that describe the
statistical characteristics of the environment in
which the system operates, including:
Environment: It is composed of four categories
(Rural, Suburban, Dense Urban, and Highrise
Urban) that difer in the concentration, density, and
height distribution of buildings. The
determination of the environment establishes the
Gaussian distribution parameters of Excessive Path
Loss, a parameter for estimating additional losses
[
That is, the specific characteristics of the channel: Transmission powers: are the power levels of the repeaters and the transmission devices of the users.
Transmission band: including the parameters of occupied bandwidth and carrier frequency. 1. The system creates a list of connections, so that each user is assigned one with which to establish the connection; 2. The system, starting from the first connection, calculates all sender-drone and drone-receiver throughputs (2* number of drones), assigning to each drone the worst of the two values, using the equation: = * 2(1 +
* )
(2) 3. Since the system is centralized, for each pair of users, the best UAV to establish the connection is chosen, and only that throughput value is considered;
Modulation: The system, modulated in OFDM, 2.3. Calculation of Throughput and Failed allows for the selection of the number of subcar- Messages: riers and their respective modulation.
Minimum rate: is the minimum throughput be- The algorithm for calculating throughput (T) and failed low which the connection fails. connections consists of two phases: 3. Properties of the UAVs and users Initialization phase
That is, those typical quantities of the movement • User and UAV positions are randomized; of users and drones, including: • The system precomputes the maximum chanVelocity: This refers to the speed of the users and nel capacity (C), given the input parameters as the UAVs. Band(B), Subcarriers(S) and Modulation (M) [25]: Number of users served and mobile repeaters: iunsdeircsapteretsheenntuinmtbheer soifmmuolabtiiloenr.adio stations and = 2 * * 2( * ) (1) Simulation area: indicates the simulation area Estimation phase considered, approximated with the base and height of the equivalent rectangle.
Robotarium developed by the Georgia Institute of Technology in collaboration with Heriot-Watt University in Edinburgh allows for the simulation of the behavior of robots and drones in a predefined area. It is possible to display the movement of users and drones in the environment as needed [24]. 4. Based on the throughput value: • if the value is equal to or greater than the maximum channel capacity (1), this is assigned as the result for the connection; • if the value is below the established threshold, the counter for failed messages is incremented; • for any other value, the connection is successful, and the calculated throughput value is assigned to the connection; 5. The simulator stores the data and starts the iteration counter, assigning each UAV a new position based on the Q-learning algorithm. 2.4; 6. At each iteration, the system checks if the drones have reached their final positions, then returns to the point 1.
Q-learning is a reinforcement learning algorithm that helps an agent learn the best actions to take in various states to maximize rewards. The Q-Learning algorithm [26] involves associating each drone with a two-dimensional matrix where rows indicate states and columns indicate actions. The system states correspond to the central position of the cells into which the drone’s operating area is divided. The actions represent the behaviors that a UAV can adopt in a given state. In the context of this article, the environment is divided into 40 states (cells of 400 * 400) while the actions available to the aircraft are 5: one for each cardinal direction and a fifth action corresponding to no movement. It should be noted that the aircraft can move a maximum of one cell per cycle.
The reward function, a parameter for updating the Q-Table, is as follows: