The paper studies the problem of assessing the preparedness of an unmanned aerial system (UAS) operator, which in recent years has become of great importance for achieving several civil and military goals. Since the use of modern technological UAS controls is characterized by a significant potential risk, the requirements for a person operating a similar system are increasing. The study offers a comparative analysis of methods for assessing the readiness of an operator to operate a UAS based on the concepts and methodology of multi-criteria optimization, which allow for taking into account some conflicting and multiple goals. The study developed a new methodology based on the operator's knowledge, skills, and psychophysiological factors. This methodology involves an automatic assessment of readiness also new users trained to operate several unmanned vehicles. It can be helpful to form an operator profile to make the right choice.
The most causes of accidents air crashes in the last decade include [
The data presented show that more than half of all aircraft accidents occur because of human errors, in most cases committed by crewmembers.
The main causes of emergencies and disasters: • violation of piloting rules, insufficient qualification of pilots for several aircraft models; • erroneous actions of the crew in difficult weather conditions; • fatigue of crew members, problems with the physical and psycho-emotional state; • ground control service errors; • poor quality of aircraft maintenance or its absence; • loss of control when entering a zone of high turbulence; • act of terrorism.
Also, catastrophes often occur in a controlled flight in a collision with the ground, which is caused by the loss of spatial orientation of the aircraft.
The conclusion that can be drawn from the analysis of the presented facts is that the causes of aviation accidents are mainly related to the activities of the pilot, who is a main element of the manmachine system.
An effective solution for the pre-flight training of a pilot is to work with a computer simulator that
is adequate for the apparatus on which he will have to perform real tasks in flight [
The simulator provides the pilot with hardware equipment and virtual means of modeling various
flight conditions. Computer simulators can be used not only to acquire specific skills necessary for
managing the aircraft but a means of assessing the level of training and advanced training of the pilot
[
The approaches which provide safe, thoughtful, and educational practices in a risk-free environment are considered. In addition, the simulator allows for the repeatability of practice exercises, can safely test different levels of training of learners, provides immediate feedback if necessary, and guarantees the performance of standardized experience for all trainees. As is known the quality of simulation situations depends on the availability of the required equipment. Flight simulators have specific equipment and are expensive equipment. For the approach to be effective and increase the likelihood of acquiring knowledge, trainees must receive immediate feedback, and the simulation must evoke realistic sensations and appropriate responses.
The most obvious purpose of using simulation training is to act in situations where the lack of knowledge and skills can lead to serious consequences. In addition, this approach to training is useful when a large number of trainees get enough practice in the workplace. The creation of an effective training system for aircraft pilots and UAV operators requires specialists in the field of situation modeling, methods of cognitive analysis, design of processes and devices, theories of automatic control and artificial intelligence (AI), as well as statistical data processing [5; 6; 8].
To date, several detailed reviews have been made that make it possible to judge the existing
methods and criteria for training personnel [3; 7; 8] in controlling the flight of an aircraft, helicopter,
or drone. The review's authors [
The criteria approach to the problem of training operators is presented in papers [
The effects of the implementation of simulator practices are presented in [
Some results of training groups of employees are presented in [5; 19 - 23]. The concept of training,
the implementation of which made it possible to identify several successful behavioral strategies that
ensure the safety of a critical object in extreme situations by a team of dispatchers, is described in the
paper [
The papers [
Assessing the pilot's readiness to perform tasks as intended is associated with taking into account
several conflicting factors obtained because of a series of measurements [
Decision-making on a set of established criteria is a systematic procedure that helps to choose the most preferable alternative possible in an uncertain situation. As a rule, there is no optimal solution to the problem of choice, and the solution depends on the decision-maker's preferences.
In contrast to the decision-making problem, multi-criteria optimization uses mathematical
optimization procedures similar to the single-criteria case. In this case, the transformation of a
multicriteria problem into an optimization problem with one criterion is performed. However, the resulting
solution to the optimization problem still depends on the parameters set by the user [
Multi-attribute Decision Making (MADM) problems with a homogeneous data type use two main approaches. In the first one, the data is reduced to a single type of input parameter, and then a decision rule is built according to the classical MADM methods. The second approach assumes mixed input parameters. In this case, the decision rule is a binary relation that allows you to select a subset of alternatives from the original set. You should expect the same result when solving the problem, but the first approach looks simpler and clearer. The most common MADM methods are considered the Simple Weighted Addition (SWA) Method, Analytic Hierarchical Process (AHP), and anticipatory methods such as Elimination and Choice Translating Reality (ELECTRE).
Solving a decision problem in AHP begins with building a hierarchical structure that includes purpose, criteria, alternatives, and other factors that influence a choice. This structure should reflect the presentation of the problem by the decision maker.
Next, priorities are determined to establish the relative importance or significance of the elements of the constructed structure, using the paired comparisons procedure. The dimensionless nature of priorities makes it possible to compare heterogeneous factors, which is a distinctive feature of AHP.
At the next stage, a linear convolution of priorities along the hierarchy is performed, because of which the priority value is determined for each alternative to the decision to the main goal. The best alternative is the one that gets the highest priority value. The final decision is made if the constructed structure meets the consistency criterion.
The elimination and choice translating reality (ELECTRE) method provides an ordering of alternatives presented in quantitative and qualitative form. The choice of an alternative is carried out according to the degree of preference. To do this, the method uses the indexes of consistency, discordance, and threshold values. The index value is in the range (0–1), which makes it possible to evaluate the reliability of each relationship and is a test indicator for each alternative. Global consistency is estimated by the Cik indicator, which confirms the consistency of all criteria under the hypothesis of the superiority of the alternative Ai over Ak. It is calculated like this = ∑ ∑ ( ) , where wj is the weight of the jth criterion; m is the total number of criteria; i, k is the estimated pair of alternatives; cj(AiAk) is the index of agreement that the alternative Ai is better than the alternative Ak in terms of criterion j. The ELECTRE method generates a system of binary superiority relations between alternatives.
The considered methods assume the presence of important data for the input parameters, which can inspire confidence in the decisions made. Different initial assumptions and constraints can lead to inconsistency in the solutions obtained by different MADM methods. To ensure the decision is made, it is proposed to use several approaches. Obtaining similar results by different MADM methods will ensure that the alternative with the highest rating is preferred because it has a high level of reliability. 3.2.
Multi-criteria optimization methods are classified according to the articulation of preferences in solving engineering problems. These include:
• Methods for a priori selection of preferences. Following these methods, it is proposed to set preferences that express the relative importance of different goals. These methods are weighted minimum and maximum, weighted product, lexicographic, linear aggregation/weighted sum, compromise programming, checkpoint, constrained objective function, desirability-based approach, goal programming, exponential weighting, etc.
• Methods for a posteriori selection of preferences. These methods are used when it is difficult to accurately determine decision function. Then it becomes expedient to choose from several already available solutions. Similar difficulties can be overcome by methods of multi-criteria optimization such as genetic algorithms, methods of normal constraints (NC), normal boundaries (NBI), intersections, and physical programming.
• Methods without highlighting preferences. Most of these methods represent some simplification with a priori preference extraction, which consists of taking the weight coefficients equal to one. Among them are the global criterion method, the minimum-maximum, the compromise function, the target sum, and the target product.
The statement of the problem of multi-objective optimization is written in the form = min ( ) = ( ( ), ( ), … , ( )) , where x = (x1, …, xm)T is the preferred solution vector. We introduce k constraints ci, ci0, i=1, ..., k, and n goal functions gj(x), j=1,…, n.
The problem is to establish an admissible set S of preferred solutions y of the vector xRm that satisfy the constraint vector G(x)Rn
= { ∈ | ( ) ∈ }, In this case, the solution y is a vector satisfying the expression = ⋃ ∈ ( ), (1) (2) (3) (4)
UAV control training requires special tools to prepare the operator for actions in real situations. An important task is to control several UAVs. The complexity of this task is characterized by the multivariance of situations requiring management decisions and the lack of wide distribution of such tools. It should be noted that the training of the operator and the assessment of his skills is still possible with the presence of a simulator based on virtual reality. In this case, the problem lies in the need to collect adequate data to compile his profile.
To this end, it is necessary to have a game engine that simplifies the learning process to control multiple UAVs. The purpose of the game engine is to create various tasks. The operator draws up a preliminary plan of action. During the plan implementation, random events lead to disrupting the successful mission completion. The operator's actions are perceived as decisions aimed at the successful completion or mission failure. The user's evaluation is based on his decision, knowledge, skills, and abilities. The information received from the simulator is interpreted in a way sufficient to evaluate the user's actions and draw up his profile. The user decisions are documented by a simulator for subsequent analysis and suggestion of corrective actions in the current situation. It is believed that the proposed approach will allow for more accurate profiling of operators. 4.1.
To create a user profile, it is necessary to have a description of the user based on measurable and understandable indicators of the simulator. To form these indicators, it is necessary to establish independent metrics. It is proposed to assess planning (PS) and monitoring (MS) skills. The first indicator shows the ability to draw up a rational plan aimed at achieving the ultimate goal, and the second indicates the ability to overcome problem situations.
A generalized indicator is also introduced, defined by a train: the ability to plan (AP), the ability to manage (AM), contact (C), and functionality (F). Here, contact C denotes the ability to maintain communication with UAVs to solve the target task, and functionality F implies the ability of the operator to correct the initial plan. The impact of the introduced indicators is provided by weight coefficients, which are in the range from 0 to 1, with 0 being the worst and 1 being the best value of the indicator. Therefore, the indicators train (AP, AM, C, F), in which each coordinate is the average value of each indicator for all user actions, characterizes the acquired user skills. 4.2.
The first step in solving a mission is to make a draft plan, which is assigned to determine the
trajectory for each UAV by defined waypoints. For example, it is done in the Mission Planner [
( ) ( ) +
( ) ( ) where W(s) is the total number of waypoints set by the user for situation s, and t(s) is the creation time of the entire mission plan. A high indicator of the values of this metric is not desirable, since further actions are possible related to the clarification of the plan and may lead to a deterioration in the overall indicator characterizing the user's skills. 4.3.
The ability to manage (AM) metric rewards the user for successful actions. The operator’s task is to be able to control the largest possible number of UAVs, return the controlled UAVs to the starting point, and minimize energy costs, including fuel, Fig. 2. (5) ( ) = (( )) + 1 − ( () ) + ( () ) where D(s) is the number of targets identified through detection, N(s) is the number of targets assigned to the current mission, U(s) are controlled UAVs, LUAV(s) is the number of UAVs lost by the user during the mission, and RUAV(s) the number of UAVs that returned at the end of the simulation. The second term, written in parentheses in formula (6), directly characterizes the efficiency of the operator's control stage in solving the target problem. In this case, LUAV(s) < RUAV(s) D(s) is assumed. (6) 4.4.
Since the user controls several devices, it is important to know the degree of his contact with the UAV during the mission. The metric is focused on measuring the number of user contacts with all UAVs, determined by the set of contacts СU = {СU1, . . ., CUN}. Here СUi is the number of user contacts with the ith UAV. This metric is proposed to be calculated as follows: ( ) = (7)
Here, is the standard deviation of various types of contacts with the UAV. If it is small, the user contacts all UAVs in the same way, and the value of C(s) tends to the maximum value, namely 1. 4.5.
The user activity is defined by his ability to function (F), i.e. his ability to correct the initial plan. For example, the operator can change the trajectory of the UAV in a mission, i.e. to correct the initial plan by introducing intermediate points, Fig.3. waypoints performed in the Monitor, Add waypoint and Manual modes respectively. Then the metric looks like this:
( ) ( ) ( ) ( ) = ( ) where W(s) = WM(s) + WC(s) + WMM(s). Weight coefficients a, b, and c are used for proper modes balancing, a + b + c = 1, c > b > a. This metric reaches its maximum value with a completely changed UAV trajectory. If F(s) is close to zero, the user practically did not change the system functionality, which is typical for a simple mission. (8)
A generalized criterion for the quality of UAV pilot simulator training is introduced, taking into account the considered parameters for evaluating user skills, which is described by the following expression
= ∑ where ai is the ith weight coefficient, i=1..4, а=(AP, AM, C, F)T, xi is the measured parameter value. The weight coefficients satisfy the identity
∑ = 1
In addition, a rating scale is introduced that determines the levels of user training Level 1, Level 2, and Level 3. These levels correspond to a high, medium, and low level of training quality according to the numerical estimates of the indicator (9). The correspondence of training levels to interval values is presented in Table. 1. (9) (10)
Following the analysis of the hierarchies process [
It is assumed that when assessing planning skills, preference is given to the ability to plan when the objects for future manipulations are motionless, and the ability to manage, contact and functionality are secondary. To evaluate monitoring skills, priority is given to alternatives that play an important role in the operator's actions when controlling moving objects, while the initial stage is secondary. Under the considered approach, weight coefficients are obtained. They are the following = 0.0625, = 0.188, = 0.312, = 0.437.
The values of the parameters obtained under the measurements carried out on the simulator and calculated by formulas (5) – (8) for three levels of training are shown in Table 2. At the same time, Level 1 corresponds to the level of initial training, and Level 3 corresponds to the highest level of UAS pilot training. (11)
The method also allows you to know the level of pilot training based on the available data. The method application scheme involves four stages. At the same time, in the first stage, it is necessary to have positive weights for each criterion. As weights, we choose the values of weight coefficients (11) obtained by the AHP.
Next, a matrix of agreement indices is constructed according to the ratio where I jk is the set of criteria by which the jth alternative is better than the kth, I jk is the set that consists of those criteria by which the alternatives jth and kth are equivalent, is a parameter that can take the values {1, 0.5, 0} depend on the method modification. In the example under consideration, the choice of this value is not of particular importance, but for definiteness, we will assume =0.5.
Diagonal values are not alternatives, and they are taken equal to unity. The matrix of agreement indices for the case under consideration has the form:
interval of superiority of the th alternative = max over the th alternative by the th criterion where the superiority interval of the alternative kth over the alternative jth by the criterion ith determines the number of consecutive transitions from class to class that must be carried out for the alternative jth to become equivalent to the alternative kth by the criterion ith, multiplied by the price one such transition. In this case, it is required the values of djk do not exceed unity. Set the transition interval for criteria AP and AM to 20 scores and for criteria C and F to 25 scores, then the disagreement matrix has the form: (13) 1 0.25 0.5 = 0.25 1 0.2
0.5 0.2 1
At the last stage, a decision rule is built, under which the final decision is made regarding the degree of preparedness. By the method, the researcher chooses the numbers p (0, 1] and q [0, 1), with the help of which a binary relation is built on the set of alternatives, establishing the superiority of the alternative jth over the kth, provided that cjk p and djk q. The choice of p and q values is made from considerations of determining the dominance of the solution, so for the values p = 0.8 and q = 0.2, it is possible to unambiguously determine the transitivity and dominance of the training levels adopted in the simulation.
The main ideological content of the work is the assessment of the operator's readiness to perform complex tasks based on cognitive and psychomotor skills obtained in the process of training on the simulator to control a group of UAVs. The methodological basis of the approach is the AHP and ELECTRE methods, which are best suited for the problem of deciding on the degree of readiness of the operator in the conditions of heterogeneity and inconsistency of the influence of the measured factors and allow compiling his training profile.
The initial data for the AHP method are the preferences put forward by the simulator developer in terms of task planning, management, contact, and operator functionality for a fixed task. The assessment of the degree of readiness is done by the formula (9), and following the results of Table 1. The approximate values of the measured parameters are given in Table. 2.
To increase confidence in the decision made, it is proposed to use the ELECTRE method, where each alternative solution is subjected to a consistency check. In this case, the weight coefficients for the AHP method are the initial ones for ELECTRE. The agreed decision is made after calculating the indices of agreement (12) and disagreement (13) and the fulfillment of the binary relations cjk p and djk q. A new aggregated decision-making approach based on the AHP and ELECTRE methods is proposed, it is a greater reliability of the decisions made.
Online assessment of learning skills is an important task for operator training in a simulated environment when working in real conditions is associated with the risk of mission failure. The value of training increases significantly in the case of managing a group of UAVs. The paper presents methods that make it possible to form a user profile and observe the growth of skills and abilities that guarantee the operator effective activity when performing actual tasks.
This work shows the possibility of using advanced methods to determine whether a user belongs to a particular group, using the developed descriptive metrics using the AHP and ELECTRE approaches.
User profiles are built on planning and monitoring abilities and skills based on measurable numbers, which are then transformed into a categorical scale. The methodology will be effective for simulators that can simulate tasks with several UAVs when the planning and monitoring tasks are solved in an integrated manner.
Further research is going to aim at expanding the possibilities of planning situations and involve
other decision-making methods based on methods of planning and processing the results of
experiments [
The work has been carried out on an initiative basis. The authors thank the anonymous reviewers, whose comments significantly improved the content of the paper.
The authors also thank both the authorities of the National Aviation University and the leadership of the Faculty of Computer Science and Technologies for their support during the preparation of this paper.