The article examines the verification of the adequacy and verification of the simulation reference model of the decision-making process at the air traffic controller of an airport traffic control tower (Tower controller) workplace when servicing arriving aircraft. The model is obtained on the basis of the previously proposed method for forming a trainee reference model an intelligent training system using the AnyDynamics software package (Rand Model Designer).
Almost all spheres of human activity are in one way or another connected with information technologies, and the level of their development often determines the success of the tasks that must be solved. One of these tasks is to realize the possibility of self-training for air traffic controllers.
At the Department of Information Technology of the Flight Academy of the National Aviation University (Ukraine), research is being carried out to improve the aircraft management quality of operators of the navigation service systems and traffic control, and the work [1] presents method with the same name, which is reflected in the intelligent training system "ATC of Tower" being developed.
The specification of this system provides the following: 1. The ability to work in the modes of demonstration, training, control, in which it is assumed, respectively: demonstration of the stages of the decision-making process when issuing permits for take-off and landing; display of special prompts that help the student in making the necessary decisions; dialogue between a student and a system that provides an opportunity to introduce independently made decisions; assessment of the student’s actions, from the point of view of quantitative and qualitative parameters of the solution formed by it. 2. Presence of the monitor of meteorological data, changing during the operation of the system, as much as possible similar to the weather display of aerodrome metrological automated system – AMAS Avia-1. 3. Availability of the aerodrome model, which reproduces the movement of aircraft along the aerodrome movement area.
The system is based on a trainee (subject of training) reference model, which in the process of functioning of the intelligent training system closely interacts with the trainee current model. As a result of their interaction, the operator's activity errors of the trainee are determined and his errors model is formed. This makes it possible to implement a mode of automatic objective control of a trainee in terms of quantitative and qualitative assessment of his qualification level and to provide him with an individual learning trajectory.
To develop a reference model, a method is proposed that includes the following stages: Stage I – data collection and knowledge extraction; Stage II – analysis and structuring of the revealed data and knowledge; Stage III – identification of regularities and formalization of the components of the reference model; Stage IV – checking the adequacy of the reference model.
This method, in the process of researching the subject area in the first three stages, makes it possible to obtain the following components, which are described in the works [2-4], for the formation of a reference model: an extended list of technological operations, the correctness of which is described by qualitative and quantitative parameters; the procedure for performing technological operations depending on the situation, namely air and ground picture (situation), aircraft performance characteristics, weather conditions, etc., with graphic visualization of the decision-making process by the air traffic controller; an information flows circulation model, developed on the basis of an analysis of the air traffic controller of the airport traffic control tower (henceforth Tower controller) workplace, for which regularities in the circulation of information have been identified; reference values of the time that is spent on performing each of the technological operations. The basis for these values is the regularities discovered among the technological operations time characteristics of the Tower controller’s activity.
For implementation of the trainee reference model the method and means of simulation modeling were used to provide the necessary high level of detail and visualization of the processes simulated in the subject area of navigation services and air traffic control.
For the simulation modeling of the reference decision-making process at the workplace of the Tower controller (the trainee reference model for the intelligent training system "ATC of Tower"), we have chosen a high-performance visual environment for the development of component models of complex dynamic systems – Rand Model Designer (from January 2021 has a new name - AnyDynamics). This software environment uses a figurative, intuitive objectoriented high-level modeling language (UML – Unified Modeling Language), which allows one to quickly and efficiently create complex models. To describe the behavior of discrete and hybrid objects, a behavior map is used – a modification of the UML state diagram, in which the activity in the state is an active dynamic object, possibly having its own internal structure [5].
Based on the results obtained at the previous stages of the study, a reference training model was formed, the structural diagram of which is shown in Figure 1, a).
The structural diagram consists of classes instances (or objects) and the relationships between them. Instances, in turn, have certain states and behavior, have certain properties (attributes) and operations performed on them (methods) [6]. A link is the connection of two external variables in a structural diagram between instances of a class. A link can connect a variable of the "output" type of one instance of a class with a variable of the "input" type of another instance. A variable of the "output" type is shown on the block diagram by an output arrow. The value of a variable of the "output" type can only be changed from inside the object. A variable of the "input" type is shown on the block diagram by an input arrow. The value of a variable of the "input" type can only be changed from outside the object. In this case, "output" variables must have a general type, and can be specified by the following scalar values: integer (int, short, char); logical (bool); string; numeric with floating-point (double), and also, the passed parameter of the external variable can be specified as an array, vector, signal and object [7].
The input data of the system are randomly generated parameters obtained on the basis of the corresponding revealed regularities. That is, at each start of the model, the values of the variables are determined by a given distribution law.
Diversification of parameters affects the variability of the air picture, which entails the need to perform various technological operations in air traffic control, changing the time of their execution, that is, it brings the situations generated by the system closer to real ones and provides a wider range of variability of these situations.
Let's consider each instance of the class in more detail.
An instance of the "Meteo" class. The main task of this instance of the class is to generate random parameters of weather conditions, as well as to determine the working runway (working course) for the entire system. The meteorological data generated by the system as a result of operation are presented in Figure 1, b).
An instance of the class "Aircraft (AC)" generates output data for the "Aircraft" class. This behavior map has a probabilistic switchpoint, which determines the presence/absence of a departing aircraft, which, in turn, depending on the position on the maneuvering area, will or will not affect the decision of the Tower controller when servicing an aircraft that arrives (approaching).
When the visual model of the system is launched, the aircraft data obtained as a result of the operation of the “Aircraft” class instance are displayed by the first two messages in the system dialog box (Figure 1, c)). The first message informs about the approaching aircraft, with the identifier "ВСз" and contains such data as: radiotelephone call sign; airport of departure; aircraft type; the wake turbulence category.
Example: ("ВСз", "UR-UBU Vienna,
BE 350 (L)")
The second message informs about the departing aircraft, with the identifier "ВСв" and contains such data as: radiotelephone call sign; destination airport; name of the route of departure from the terminal area (SID – standard instrument departure); aircraft type; wake turbulence category; aircraft parking position (stand).
Example: ("ВСв", "MSI740, Tbilisi, DITIX4A,
An-74 (M), St. No. 41")
An instance of the "Ground Services (GS)" class generates data for aerodrome services supporting activities at the aerodrome through the assignment of a radiotelephone call sign and a possible request for the runway crossing. Interaction is also presented in the form of radiotelephony exchange in a dialog box.
"Briefing" class instance simulates the interaction of the Tower controller and the briefing office (ARO) dispatcher – transmitting information to the Briefing Office dispatcher about the actual time (UTC) of aircraft takeoff/landing, as well as receiving information about the aircraft departure from another aerodrome.
An instance of the class "Aerodrome Central Dispatcher (ACD)". The main task of this class in the simulated model is to request / receive information from the ACD about the parking for the arriving aircraft and transmit information to the ACD about the actual time (UTC) of the aircraft landing.
An instance of the "Approach Controller" class. In addition to the main task, the transmission of information about the approach type of the arriving aircraft to the aerodrome area, this instance of the class simulates the interaction and coordination between the controllers of the approach sector and the aerodrome control tower in the event of a missed approach of this aircraft – imitation of an unsuccessful approach.
An instance of the "Tower Controller" class is a model of the decision-making process by the Tower controller, which simulates the actions of the controller when servicing an approaching aircraft. That is, in this class, the technological operations of the Tower controller are concentrated during direct work with the aircraft crews. When the visual model of the system is launched, the work of this instance of the class is displayed in the form of radiotelephony phraseology (in Russian) between the air traffic controller and the aircraft crew in the dialog box (Figure 1, c)).
In addition to the "Aerodrome Plan" window, which displays the chart of the simulated aerodrome, the visual model also has a "Time" window (Figure 1, d)), which displays the time of the following procedures (sets of technological operations): 1. Procedure "1-1'" – transmission of the information about the aircraft departure from another aerodrome by the Briefing Office dispatcher. 2. Procedure "2-2'" – the Approach controller sends information about the approaching aircraft. 3. Procedure "3-3'" – the final stage of the aircraft approach. At this stage, the controller makes a decision on issuing a landing clearance in accordance with the air picture at the aerodrome (available aircraft for departure, work on the runway) and its area (other aircraft is going-around). 4. Procedure "4-4'" – vacating the runway after landing and taxiing the aircraft to its parking position. 5. Procedure "5-5'" – transmission of information to the Briefing Office dispatcher about the actual time (UTC) of aircraft landing.
The adequacy of the model was checked using the above time parameters and time parameters of the real system. 3.
To ensure the appropriate accuracy and reliability of the simulation results, it is necessary to check the adequacy and/or verification of the model. The purpose of these procedures is to establish identity in a certain sense (in terms of goals, functions, tasks, operations, static and dynamic parameters, indicators, etc.) of a model and a real object, or to establish the identity of two models.
The verification of the adequacy of the simulation reference model when servicing arriving aircraft, from the side of the qualitative criterion, was carried out both at the stage of constructing a formalized scheme of the process (the algorithm of actions of the Tower controller), and at the stage of its computer implementation (when the dynamic model is functioning, a logical and procedurally correct radiotelephony communications between subscribers and the air traffic controller). Checking the adequacy of the resulting model in relation to quantitative indicators can be performed using formal and informal methods [8].
Verification using methods of statistical analysis refers to formal methods. It is possible with reliable statistical estimates of the parameters of both real operations of the air traffic services system and the model. In fact, two independent groups of time characteristics data (real object and model) performed by the Tower controller of procedures, consisting of a set of corresponding technological operations, for which their inherent regularities were revealed at the previous stage of the study.
To find out which of the criteria can be used to assess the adequacy, the analysis of time indicators of the execution of procedures (in seconds) of the simulation model (scan data) and the real system (timing data) was carried out using the descriptive statistics method (Table 1).
The normal distribution is determined depending on the fulfillment of certain criteria. One of these criteria is the coincidence of the average (mean), thickest value and median. The skewness, in turn, characterizing the normal distribution should be in the range from –1 to +1, and sometimes a distribution with a skewness not exceeding 2 in modulus is considered normal one [9]. Another important criterion is kurtosis. It is believed that a distribution with kurtosis in the range from –1 to +1 corresponds approximately to the normal form. Sometimes it is quite acceptable to consider a distribution as normal with kurtosis in absolute value not exceeding 2 [9].
The results obtained allow us to conclude that the distribution of data for Procedures "1-1", "22", "5-5" and "6-6" can be attributed to normal, and, therefore, a parametric method for comparing quantitative data in two independent groups test - Student's t-test.
When comparing the mean values in normally distributed sets of quantitative data, the Student's t-test is calculated by the formula [10]: = where: M1 and M2 are the compared average (mean) values, m1 and m2 are the standard errors of the average (mean) values, respectively.
The obtained values of the Student's t-test are evaluated by comparison with the critical values. Differences in indicators are considered statistically significant at a significance level of p<0.05 [10].
Based on the results obtained (Table 2), we can say with a high degree of probability that the differences between the simulation model and real data are not significant (Procedures "1-1", "6-6"). Moreover, the differences continue to be insignificant even with an increasing sample size (for Procedures "2-2", "5-5" the number of degrees of freedom increases by almost a third), which indicates the adequacy of the real system of the quantitative component of the resulting model and its resiliency.
As for the Procedures "3-3", "4-4", the use of Student's t-test is not recommended due to the fact that their temporal characteristics do not agree with the normal distribution. In this case, it is possible to use the Mann-Whitney U-test. For this, a single array of both compared samples is compiled, their elements are arranged according to the degree of growth of the feature, a lower value is assigned a lower rank. Then a single ranked series is divided into two, consisting of units of the first and second samples, in each of which the sum of the ranks is calculated separately. After that, the value of the U-criterion is calculated according to the following formula [11]: = 1 ∙ 2 + ∙ ( + 1) − 2 (2) where n1 is the number of elements in the first sample, n2 is the number of elements in the second sample, nx is the number of elements in the larger sample, Tx is the sum of the ranks in the larger sample.
The calculated values of the Mann-Whitney Utest are compared with the critical values at a given significance level: if the calculated U-test value is equal to or less than the critical U-test value, the statistical significance of the differences is recognized. Verification of two independent groups of data (the model and the real system) for Procedures "3-3" and "4-4" with a different number of time characteristics (15 and 20 values) using the Mann-Whitney U-test (Table 3) showed that differences in the level of the feature in them are statistically insignificant (р>0,05), which indicates the adequacy and sufficient stability of the model.
Verification is a determination of the correctness of a developed program, formal or practical proof of its correct operation on a computer [12]. For additional verification of the adequacy of the obtained model of this kind of dynamic stochastic system, a direct method of model verification is selected – verification by developing a model of the same object (its parts) using another mathematical method.
An alternative mathematical method is the GERT (Graphical Evaluation and Review Technique) critical path method. If we compare the statistical parameters of the average (mean) and standard deviate of the time characteristics of 72 > 64 – differences in the level of the feature in the compared groups are statistically insignificant 170 > 127 – differences in the level of the feature in the compared groups are statistically insignificant 65 > 64 – differences in the level of the feature in the compared groups are statistically insignificant 159 > 127 – differences in the level of the feature in the compared groups are statistically insignificant the procedures of the three sources (Table 4) – the simulation model, the timing data and the parameters of the temporal characteristics obtained on the basis of the use of GERT – we can also conclude that the difference between these indicators varies within the limits one second.
For procedures "3-3 '" and "4-4'", analytical calculations to determine the moments of the distribution function of the output quantity using GERT networks were not carried out, due to the presence of subsystem blocks in these procedures (the final stage of the aircraft approach and the vacating of the runway after landing and taxiing of the aircraft to its parking position), the execution time of which is directly proportional to the flight technical characteristics of specific aircraft, and the calculation of distribution parameters becomes possible only with simulation modeling.
The performed verification of the adequacy of the simulation reference model of the decisionmaking process at the workplace of the Tower controller using formal statistical criteria and its verification allow us to take up the position that the model is adequate and sufficiently reflects the real system, which avoid the necessity for its adjustment. Satisfactory results obtained at this stage of the study also make it possible to make a positive conclusion about the efficiency of the proposed method for forming a reference model of an intelligent training system and the feasibility of its further use.
5. References [1] O. M. Piliponok, Method of improving the aircraft management quality of operators of the navigation service systems and traffic control. The thesis for a candidate of technical science degree in speciality 05.22.13 «Navigation and traffic control». – Separated structural unit of the National Aviation University «Kirovograd flight academy of the National aviation university», Ukrane, Kropyvnytskyi, 2017, 265 p. (in Ukrainian). [2] L. N. Dzhuma and A. S. Lavrynenko, Detalyzatsyia tekhnolohycheskykh operatsyi dyspetchera ADV s yspolzovanyem metoda khronometrazha. Upravlinnia vysokoshvydkisnymy rukhomymy obiektamy ta profesiina pidhotovka operatoriv skladnykh system. Kirovohrad: KLA NAU, 2015, pp. 195–196. URL: http://www.klanau.kr.ua/images/docs/mmnp k2627112015.pdf [3] L. N. Dzhuma and A. S. Lavrynenko, Unyfykatsyia tekhnolohycheskykh operatsyi deiatelnosty avyadyspetchera pryrazrabotke etalonnoi modely subъekta obuchenyia. Aviatsiia ta kosmonavtyka: stan, dosiahnennia i perspektyvy, Part 1. Kirovohrad: KLA NAU, 2016, pp. 201–202. URL:http://www.klanau.kr.ua/images/docs/ mvnpkxxxvip01.pdf. [4] L. N. Dzhuma and A. S. Lavrynenko, Vyyavlenie zakonomernostey tsirkulyatsii informatsionnykh potokov na rabochem meste dispetchera aerodromnoy dispetcherskoy vyshki Tower. Upravlinnia vysokoshvydkisnymy rukhomymy obiektamy ta profesiina pidhotovka operatoriv skladnykh system. Kirovohrad: KLA NAU, 2016, pp. 94–96. URL: http://www.klanau.kr.ua/images/docs/5mnp k.pdf [5] Yu. B. Kolesov and Yu. B. Senichenkov, Ob’ektno-orientirovannoe modelirovanie v srede Rand Model Designer [Object-oriented modeling in the Rand Model Designer environment]. SPbPU Science Week: Materials of the Scientific Forum from Intern. participation. Interdisciplinary sections and plenary sessions of institutes. St. Petersburg: Polytechnic University Publishing House., 2015. pp. 18-25 (in Russian). [6] G. Buch, Obektno-orientirovannyj analiz i proektirovanie s primerami prilozhenij na C++ [Object Oriented Analysis and Design with Sample C ++ Applications] Part 1, Vol.2, Moscow: Binom, 1998, 560 p. (in Russian). [7] AnyDynamics – high-performance visual environment for object-oriented modeling of multi-domain component models. MvStudium. [Electronic resource]. URL: http://www.mvstudium.com. [8] N.G. Lipatova, Imitatsionnoe modelirovanie protsessov tamozhennogo kontrolya: monografiya [Simulation modeling of customs control processes: monograph]. Moscow: Publishing house of the Russian Customs Academy, 2015. 164 p. (in Russian). [9] E. V. Dorogon'ko, Obrabotka i analiz sotsiologicheskikh dannykh s pomoshch'yu paketa SPSS [Processing and analysis of sociological data using the SPSS package], Surgut: Publishing Center of SNU. 2010, 60 p. (in Russian). [10] J.L. Peacock and P.J. Peacock, Oxford Handbook of Medical Statistics. Oxford University Press, 2011. 517 p. [11] M.J. Campbell, D. Machin and S.J. Walters, Medical statistics: a textbook for the health sciences. 4th ed, John Wiley & Sons, Ltd., 2007, 331 p. [12] I.V. Yatskiv, Problema validatsii imitatsionnoy modeli i ee vozmozhnye resheniya [The problem of validation of the simulation model and its possible solutions]. Simulation modeling. Theory and Practice (IMMOD 2003): Collection of reports of the First All-Russian Scientific and Practical Conference on Simulation Modeling and Its Application in Science and Industry, Part 2. St. Petersburg: Federal State Unitary Enterprise "Central research institute of shipbuilding technology", 2003, pp. 211217. (in Russian).