Methodology for Evaluation the Performance Indicators of the Ergatic Information System Functioning Tetiana Vakaliuka, Ihor Pilkevychb, Andrii Tokarb and Roman Lobodab a Zhytomyr Polytechnic State University, Chudnivska str., 103, Zhitomir, 10005, Ukraine b Zhytomyr military institute named after S.P. Korolov, Mira ave., 22, Zhitomir, 10004, Ukraine Abstract The paper proposes a generalized methodology for evaluation the performance indicators of the ergatiс information system, which is based on an algorithmic model of the operator's activity and takes into account its reliability. For this purpose, the process of operator activity is represented in the form probabilistic graph, which is a graphical representation of individual operations and allows to describe the algorithm mathematically, to evaluate its hourly and probabilistic indicators using the rules of transformation graph-scheme. Proceeding from the purpose of the ergatiс information system, the tasks that it solves, the indicators are selected the effectiveness of its operation. The order of their calculation is given. The influence exerted by the operator reliability on the efficiency performance of the ergatiс information system is taken into account by means a coefficient which characterizes the operator's ability to work. By the example radio-monitoring system calculations indicators efficiency of its functioning are carried out. Keywords 1 Coefficient the reduction of operator reliability, ergatic system, operator reliability, performance indicators, system functioning performance. 1. Introduction The efficiency of functioning within any ergatiс information system depends on the reliability of its components: technical means and operator. The modern science allows us to establish successfully the laws of occurrence failure occurrence of devices and methods to forecast them, to find methods to improve reliability during their design and the following manufacturing, as well as methods and techniques to maintain reliability during their storage and operation. At the same time, the individual nature and high variability of human psychological, physiological and professional capabilities and characteristics, its sensitivity to the influence exerted by external and internal environment factors complicates the processes related to analysis, forecasting and improvement of human-operator reliability. As a consequence, due to human errors as a result of her insufficient training, unfavorable psychological factors, fatigue occur the majority share in all the accidents and accidents in different branches of activity [1-3]. Under such conditions, the issue relating to the evaluation indicators of the effectiveness at functioning ergatiс information system is topical. Under such system we will understand the interaction of technical means and the operator, whose activity is aimed at the timely identification and processing significant volume of information flows, rapid analysis of the received data, development an optimum decision concerning the further actions. Examples such activities are the actions of an air traffic controller, operator unmanned aircraft complex, operator of a radio monitoring post and the like. CMIS-2021: The Fourth International Workshop on Computer Modeling and Intelligent Systems, April 27, 2021, Zaporizhzhia, Ukraine EMAIL: tetianavakaliuk@gmail.com (T. Vakaliuk); igor.pilkevich@meta.ua (I. Pilkevych); tapir@i.ua (A. Tokar); romaloboda0704@gmail.com (R. Loboda) ORCID: 0000-0001-6825-4697 (T. Vakaliuk); 0000-0001-5064-3272 (I. Pilkevych); 0000-0001-7534-2820 (A. Tokar); 0000-0003-4010- 0252 (R. Loboda) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) To prevent a failure in the execution of tasks, it is necessary to evaluate the performance indicators of the information system in the process of its operation and compare their values with the limiting value. Based on this, there is an urgent need to calculate these indicators in real time, taking into account the peculiarities of the operator's activities in specific conditions, in order to prevent the deterioration of the quality of tasks performed by the system as a whole. To do this, we will formalize and simulate the process of the system functioning, as a result of which we will obtain the dependence of the performance indicators of the information system on the operator's reliability. When constructing a model, we will determine its qualitative composition and numerical parameters. 2. Review of the literature Sufficiently well-developed approaches to the evaluation efficiency of systems at the stage of their design [4, 5]. Estimation a degree of growth probability human error depending on changes of environmental conditions using prediction model is described in [3], but its influence on the efficiency the system is not shown. Impact that system reliability and task uncertainty have on unmanned aerial vehicle operator performance during patrolling and target recognition tasks is investigated in [6], but the inverse effect is not explored. The issue concerning quantitative estimation probability of human error on the basis the qualitative analysis human factor and task context with additional consideration of factors forming performance is devoted to the research work [7]. In research [8] measurement from the operator's reliability level is carried out using Human Error Assessment and Reduction Technique (HEART) on the example of 3 male operators using computer numerical control milling with work experience > 3 months, operator age between 18 and 39 years. Method HEART is proposed to be applied to analyze human realibility in production by is done using Hierarchical Task Analysis in research [9]. In [10] the HEART method is used in order to quantify the human failure event for probabilistic safety assessment of a TRIGA Mark II reactor, situated at the Malaysia Nuclear Agency, Malaysia. This approach requires the selection values nominal probabilities of human error and does not take into account the time factor. A mutual dependence between machine degradation and human error in man-machine systems are studied in research [11], but the influence of other factors on the efficiency of the system functioning is not taken into account. In [12], it is proposed to evaluate the efficiency of the system using some quantitative parameters such as servicing time and costs. This approach does not take into account the degree of decrease in the reliability of the system user. The issue of professional suitability at the stage of selection of candidates for training pilots of the United States Air Force MQ-1 Predator is investigated in [13]. The study of changes in the efficiency of the MQ-1 Predator depending on the reliability of the operator in this work is not performed. A number of works are devoted to assessing and improving the reliability and efficiency of systems functioning, but at the same time, no attention is paid to the influence of operator reliability on the performance indicators of the system as a whole [14-16]. Analysis shows that quite successfully solved the issue concerning the evaluation of the effectiveness systems at the stage their design. At the same time, during the assessment and improvement of the reliability and efficiency of the systems functioning, attention is not paid to the influence of the operator's reliability on the performance of the system as a whole. In the direction of evaluation efficiency systems in their functioning studies conducted limited to the assessment a person's reliability without calculating the performance measures of the system as a whole, in addition is not taken into account the factor changes in the probability or human errors over time. So, the purpose the work is to develop a methodology for evaluation the performance indicators of the ergatiс information system, taking into account the reliability of the operator. 3. Research methods The functioning of the ergatic information system will be considered as a process of timely identification, receipt and processing of information flows by the operator, rapid analysis of the data obtained, the development of optimal solutions for further action. In this case, we will assume that the technical means of the information system work faultlessly at a given time interval. Under such conditions, the effectiveness of the functioning of the ergatiс information system will be determined by the timeliness and validity of decision-making by the operator. The probability of a certain event is used as performance indicators in conditions of uncertainty [3, 6–8]. Therefore, as indicators of efficiency of functioning of ergatic information system we use probability of timely decision making – Pt and probability of error-free decision making – Pd by operator of information system. 3.1. Method for evaluation the probability of timely decision-making The probability of timely decision-making by the operator of information system – Pt characterizes the probability of event Tmin  Po  Ta , that is P  P(T  P  T ), (1) t min o a where Tmin – is the minimum possible time of the operator's decision; Ta – is the maximum allowable time of the operator's decision; To – time is the operator's decision. According to probability theory [17] P(Tmin  Po  Ta )  F(Ta )  F(Tmin ) , (2) where F(Ta ) – is the time distribution function Ta ; F(Tmin ) – is the time distribution function Tmin . Due to the influence of a large number and random variables on the information system operator (increase or redistribution efforts from information processing bodies when the situation changes; constant influence on the operator the results from the information flows processing; failures in the regulation process and reporting the data received, decrease in personnel training, deterioration of communication channels quality, quality level of information base used; lack as well as inadequate resources to implement processing in case of complicated situation on the information system operator's side) there is every reason to believe that the time for making a decision by the operator is distributed according to the normal law [17]. In such a case the density of the distribution function of the operator's decision-making time will be equal to ( t  m ) 2 1 2 2 (3) f (t )  e  2 . From expression (3) we obtain the time distribution functions Tmin and Ta ( Tmin  m )2 1 Tmin 2 2 (4) F ( Tmin )  e dTmin ,  2   ( Ta  m )2 1 Ta 2 2 (5) F ( Ta )  e dTa ,  2   where  – is the standard deviation of the operator's decision time; m – is the mathematical expectation of the operator's decision time. Expressing the distribution functions (4) and (5) through the standard distribution function (6) ( x ) 2 1 x 2 (6) Ф( x )  e dx ,  2   using expressions (1) and (2) we obtain T m T m Pt  P(Tmin  Po  Ta )  Ф( a )  Ф( min ). (7)   Over time, as a result for various reasons, when an operator makes a decision, and hence the mathematical expectation time of making a decision will increase and is determined by the dependence m  m  t , where t – is the value of the increase in the average time of the operator's decision due to a decrease in its reliability t  ( Tmax  m)K R , (8) where K R – is a coefficient characterizing the reduction of operator reliability. In the optimal state operator – K R  0 , in a critical decrease in the reliability operators K R  1 . Procedure for calculation of K R is given in [18]; Tmax – time for decision making at a low level of reliability the operator of the information system, i.e. when K R  1 . Taking into account (7), (8), we obtain an expression for the evaluation of the probability of timely decision making by the information system operator at any point of time T  ( m  t ) T  ( m  t Pt  Ф( a )  Ф( min )   (9) Ta  Tmax K R  m( 1  K R ) Tmin  Tmax K R  m( 1  K R  Ф( )  Ф( ).   To calculate the probability of timely decision-making by the operator at an arbitrary moment using expression (9), it is necessary to know the minimum possible decision-making time – Tmin , the maximum possible decision-making time by the operator – Ta , average decision-making time by the operator in optimal conditions – m , decision-making time at a low level operator reliability – Tmax . For definition of mentioned parameters, we will use method on the base an algorithm probability graph [19, 20], which is based on analysis of algorithm of operator's activity. For the effective usage algorithmic models, it is necessary to obtain reliable data on the elements of operator activity, i.e. to analyze the functioning process in a particular information system. As a result, to establish the time, probabilistic characteristics and accuracy indicators of the information system operator's activity, based on the data given in the reference literature and their clarification, taking into account various factors (operator's state, ergonomic characteristics of the system etc.). For example, let us calculate the probability of a timely decision by the operator of the radio monitoring system. For this, an analysis of the functioning of such a system based on a radio receiver was carried out. As a result, based on the list of operations and logical conditions that were obtained by the expert method, a structural diagram of the activity of the operator of the radio monitoring information system was drawn up (Figure 1). A list of operations of the operator's activity algorithm has been compiled and the time indicators of each operation are given – Tmini , Ta i , mi , Tmaxi (Table 1). Understanding Deciding on the Receiving radio Registration of data the task operating mode monitoring data on tangible media Accounting for the Primary study Analysis Generalization received data Report Figure 1: Block diagram of the activity of the operator of the radio monitoring information system Table 1 Operator activity operations and their timing № Tmini , c Ta i , c Tmaxi , c Operation name mi , c operation 1 Understanding the task 25 50 30 60 2 Deciding on the operating mode 1,2 3 1,8 5 3 Switching the receiver to radio monitoring mode 5 10 7 15 4 Switching the receiver into search mode 5 10 7 15 5 Signal processing decision making 1,5 3 2 4 6 Selection of the signal of interest 3 6 4 8 7 Pre‐latching (signal acquisition) 5 10 7 12 8 Preliminary analysis of the information message 10 25 15 32 9 Determination of signal novelty 5 25 10 32 10 Sending a direction finding command 2,3 4,3 2,8 6 11 Getting results direction finding 2 5 3 7 12 Assignment of bearing to a specific source 5 10 7 16 13 Establishing the main features of the source 60 120 80 130 14 Source identification 1,5 3 2 5 15 Registration of data on tangible media 50 110 70 120 16 Familiarization with data content 60 160 85 174 17 Highlighting known and unknown elements 10 30 17 50 18 Determining the importance of a new element 3 7 4 11 19 Determination of urgency 3 7 4 11 20 Report 10 20 11 30 21 Validity informations assessment 30 70 40 90 22 Determining the informations characteristics 40 80 50 110 23 Comprehensive data exploration, comparison with available data 120 240 150 280 24 Identifying changes in condition and character 5 15 8 17 25 Formation of conclusions 130 270 170 340 26 Formation of information documents 180 400 220 420 27 Thematic systematization 25 45 30 70 28 Accounting for the received data 120 250 170 310 29 Results report 10 20 11 30 Time indices Tmin , Ta , m , Tmax we will receive on the basis of time indices individual operations Tmini , Ta i , mi , Tmaxi (Table 1) by representing the algorithm and using the rules of its transformation [19, 20]. Probabilistic oriented graph, corresponding to the algorithm of radio monitoring system functioning is shown in Figure 2. It is a graphical representation of individual operations and allows to describe the algorithm mathematically, to estimate the time and probabilistic indicators. Each vertex of the graph corresponds to an operation according to the numbers from Table 1 and a 4-dimensional vector Vi , whose components are Tmini , Ta i , mi , Tmaxi . The set graph vertices and its branches form the graph-scheme of the algorithm. To obtain time indicators in the algorithm we use the rules of graph-scheme transformation [20]. 3 5 1 2 7 8 9 10 11 12 4 6 13 14 15 16 29 28 27 26 25 24 23 17 22 21 19 18 20 Figure 2: Probabilistic directed graph 1. Combining paths without branches. The path graph without branches has the form (Figure 3): 1 2 …. n V1 V2 Vn Figure 3: Graph without branches The concatenation operation turns a branch without branches into an equivalent branch with one vertex (Figure 4). 1-n Ve Figure 4: Equivalent branch with one vertex The components of the vector Ve are found by the expressions: n Tmin e   Tmini i 1 n Ta e   Tai i 1 (10) n me   mi i 1 n Tmax e   Tmaxi i 1 2. Combining paths with a branching. The graph of a path with a branching has the form (Figure 5): p 2 V2 1 4 V1 V4 1-p 3 V3 Figure 5: Graph of a path with a branching The concurrent paths joining operation turns a branch with a branching into an equivalent branch with one vertex with vector Ve whose components are found by the expressions: Tmin e  Tmin1  pTmin2  ( 1  p )Tmin3  Tmin4 ; Ta e  Ta1  pTa2  ( 1  p )Ta3  Ta4 ; (11) me  m1  pm2  ( 1  p )m3  m4 ; Tmax e  Tmax1  pTmax2  ( 1  p )Tmax3  Tmax4 , where p – is the probability of transition along a branch, taken as 0,5 for all branches. According to the given rules let us join paths without branches into graph (see Figure. 2) (1, 2), (3, 5), (4, 6), (7, 8, 9), (10, 11, 12 ), (13, 14, 15, 16, 17), (18, 19), (21, 22), (23, 24), (25, 26, 27, 28, 29). As the result we obtain a probabilistic oriented graph presented in Figure 6. 18-19 20 10-12 3-5 1-2 7-9 13-17 21-22 4-6 23-24 25-29 Figure 6: Result of consecutive branch merging Using expressions (10), we obtain equivalent time characteristics of graph vertices (Table 2). Table 2 Time characteristics after combining № operation Tmini , c Ta i , c mi , c Tmaxi , c 1‐2 26,2 53 31,8 65 3‐5 6,5 13 9 19 4‐6 8 16 11 23 7‐9 20 60 32 76 10‐12 9,3 19,3 12,8 29 13‐17 131,5 313 184 359 18‐19 6 14 8 22 20 10 20 11 30 21‐22 70 150 90 200 23‐24 125 255 158 297 25‐29 465 925 601 1170 To combine paths with branching, accept the same probability to go through each branch of the logical operator is p  0 ,5 . As a result combining paths with branching (1-2, 3-5, 4-6, 7-9) and (18- 19, 20, 21-22) the graph-scheme will look like (Figure 7). 10-12 18-22 1-9 13-17 25-29 23-24 Figure 7: Result of consecutive branch merging Using expressions (11), we obtain equivalent time characteristics of graph vertices (Table 3). Table 3 Time characteristics after combining № operation Tmini , c Ta i , c mi , c Tmaxi , c 1‐9 53,3 127,2 73,6 161,6 10‐12 9,3 19,3 12,8 29 13‐17 131,5 313 184 359 18‐22 81 174 103,5 237 23‐24 125 255 158 297 25‐29 465 925 601 1170 Combine the paths with branching (1-9, 10-12, 13-17) (Figure 8) and obtain the equivalent time characteristics of the vertices of the graph (Table 4). 18-22 1-17 25-29 23-24 Figure 8: Result of consecutive branch merging Table 4 Time characteristics after combining № operation Tmini , c Ta i , c mi , c Tmaxi , c 1‐17 187,59 446 261,4 529,3 18‐22 81 174 103,5 237 23‐24 125 255 158 297 25‐29 465 925 601 1170 After combining paths with branching (1-17, 18-22, 23-24, 25-29) (Figure 9), we obtain equivalent time characteristics of the equivalent graph vertices (Table 5). 1-29 Figure 9: Result of consecutive branch merging Table 5 Time characteristics after combining № operation Tmini , c Ta i , c mi , c Tmaxi , c 1‐29 818 1626 1072 1996 Obtained time indices algorithm activity of the radio monitoring system operator gives the opportunity to find the probabilities of timely decision making by the operator at any point in time by expression (9). Taking into account that in solving the problem the operator has normal law distribution, the average deviation in time of taking the decision by the operator will be found by expression  m  Tmin  3 . (12) The obtained value of the probability of a timely decision-making by the operator must be compared with the admissible value. The permissible value can be obtained in further research in relation to a specific task performed by the system. So, the method for evaluation the probability of timely decision-making by the information system operator will consist of the following steps: 1. Make a probabilistic oriented graph of the functioning of the ergatiс information system. 2. Determine the timing of individual operations – Tmini , Ta i , mi , Tmaxi . 3. Perform transformations of the graph using the rules of combining paths without branches and with branches and calculate the time indices – Tmin , Ta , m , Tmax . 4. Calculate the average deviation of the operator's decision time by expression (12). 5. Calculate the probability of timely decision-making by the operator of the information system by expression (9). 3.2. Method for evaluation the probability of error-free decision-making To calculate the probability of error-free decision making by an information system operator we will use a method based on information analysis of decision making processes [21, 22]. The basis of the informational approach is the fact that the most essential factor, which influences the error-free decision making, is the volume of initial information. Accordingly, for any complex system, an increase in the amount of initial information leads to an increase in the probability of error-free decision making by an operator. Taking into account loss of initial information due to decrease of operator's reliability, let us write the expression for probability of error-free decision – Pd of the i -th operation: ( I in  I l )  Ki (13) Pd i  1  Y0 e , where I in – amount of initial information; I l – amount of information lost; Y0 – initial uncertainty of the decision taken characterizes the psychological confidence of the operator (at the optimal state of the operator Y0  1 ); K i – coefficient that characterizes the value an information and affects the growth rate of probability depending on the amount of processed information by the operator. At maximum value of the information K i tends to 0, at minimum – K i = 1. The value of this coefficient will be found on the assumption that as the number of possible states of the system, and consequently the number of variants of decisions n , grows, the value of the information by the expression 1 Ki  . (14) n The amount of initial information about some system is equal to the entropy of this system I in  N , and at equal-probability n states of the system [17] I in  log n. (15) Over time, due to the impact of various reasons, part the original information will be lost, the value of the loss will depend on the level efficiency of the operator and determined by the introduced coefficient decrease in the reliability of the operator – K R . The amount is lost information we will find by the expression. I l  I in .K R . (16) The level of reliability will also affect the level in psychological confidence, that is, the coefficient Y0 will decrease in accordance with the value of K R . With the decrease of the level a reliability operators value of information will decrease. To take this into account let's write expression (14) in the form: K Ki  R . (17) n Given expressions (13), (15), (16), (17), and assuming that the operator solution is completely indeterminate is Y0 =1, we obtain an expression for determining the probability of error-free solution at the i -th operation at an arbitrary time moment: n ( 1 K R ) log ni  i KR (18) Pd i  1  K R e . To carry out calculations on the example the radio monitoring system, by expert polling was established the number from the i -th operation, determining the number of states – n (Table 6, Table 7). In the table the numbers of operations correspond to the numbers from Table 1. Table 6 Time characteristics after combining № operation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Number of 3 2 2 2 2 4 4 13 4 4 4 4 14 7 9 possible solutions Table 7 Time characteristics after combining № operation 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Number of possible 9 16 16 16 2 16 10 9 16 2 2 4 5 2 solutions To obtain the probability for error-free decision by the operator of the radio-monitoring system, was used probabilistic directed graph, represented on Figure 2. In the process of transforming the initial graph-scheme to the equivalent one, with one vertex, the union of the same paths was carried out, as in determining the time characteristics, taking into account the rules for transforming the graph [20]. It was taken into account that when merging paths without branching (Figure 10) 1 2 …. n Pd1 Pd2 Pdn Figure 10: Paths without branching probability characteristics are determined by the expression [20] n Pd   Pd i . (19) i 1 When combining tracks with branching (Figure 11) p 2 P2 P1 P4 1 4 P3 1-p 3 Figure 11: Paths without branching probability characteristics are determined by the expression Pd  P1P4 ( pP2  ( 1  p )P3 ). (20) So, the method for evaluation the probability of error-free decision-making by the information system operator will consist of the following steps: 1. Make a probabilistic oriented graph for functioning of an ergatiс information system. 2. Determine the number or possible solutions of the i -th operation determining the number of states – n . 3. Calculate the probability for error-free decision by the operator when performing the i -th operation at an arbitrary moment of time by expression (18). 4. To carry out transformations to the graph using the rules of combining paths without branching and with branching and to carry out calculation – Pd by expressions (19), (20). In order to verify the proposed methods for evaluation the performance of the ergatic information system, the probabilities involved in the timely decision-making and the probability of error-free decision-making by the operator information system for the range of changes in the coefficient K R  0  1 were calculated. The results are shown in Table 8, for which the corresponding dependencies were built (Figure 12). Table 8 Calculation results KR Pt Pd 0 1,00 1,00 0,1 0,99 0,99 0,2 0,99 0,99 0,3 0,99 0,99 0,4 0,94 0,97 0,5 0,81 0,85 0,6 0,58 0,50 0,7 0,28 0,13 0,8 0,05 0,02 0,9 0,01 0,01 1 0,00 0,00 Pd , Pt Pd Pt KR Figure 12: Dependence of indicators on K R Analysis of the data obtained shows that at the initial stage of reducing the operator's reliability (up to K R = 0.4), the performance indicators of the system functioning practically do not change. This can be explained by the inclusion of human compensatory mechanisms. After a certain value, a sharp decrease in the performance indicators of the system functioning to critical values is observed. 4. Conclusions The developed methods constitute the content of the generalized methodology for evaluation the performance indicators for ergatic information system, based on an algorithmic model of the operator's activity. The methods take into account the reduction factor of operator reliability, made it possible to obtain an estimate of probabilities for timely decision-making and probability of error-free decision-making by the information system operator in real time while on duty. 5. References [1] T. Kletz, P. Amyotte, Accidents said to be due to human error, What Went Wrong?, 6th. ed. Butterworth-Heinemann. (2019) 53–172. doi:10.1016/B978-0-12-810539-9.00007-0. [2] B. Strauch, Investigating Human Error: Incidents, Accidents, and Complex Systems, 1st. ed., Routledge (2018) doi:10.4324/9781315251851. [3] Y. Tan, D. Feng, H. Shen, Research for Unmanned Aerial Vehicle components reliability evaluation model considering the influences of human factors, 3rd International Conference on Mechanical, Electronic and Information Technology Engineering., ICMITE 2017, vol. 139, article number 00221. doi:10.1051/matecconf/201713900221. [4] Z. Mian, D. Mavris, Mission effectiveness quantification and assessment of Micro Autonomous Systems and Technologies, IEEE Systems and Information Engineering Design Symposium (2013) doi:10.1109/SIEDS.2013.6549514. [5] Haifeng Zhu, Fundamental Models for Missions of Engineered Systems, Systems Conference (SysCon) IEEE International. (2019) 1–6. doi:10.1109/syscon.2019.8836714. [6] D. Liu, M. Jaramillo, D. Vincenzi, The effects of system reliability and task uncertainty on autonomous unmanned aerial vehicle operator performance under high time pressure. Hum. Factors Ergon. Vol. 25, № 5. (2015) 515–522. doi:10.1002/hfm.20565. [7] X. Pan, Z. Wu, Performance shaping factors in the human error probability modification of human reliability analysis, International Journal of Occupational Safety and Ergonomics, Vol. 26. (2020) 538–550, doi:10.1080/10803548.2018.1498655. [8] Y. Rejeki, E. Achiraeniwati, A. Wanda, Measurement of operator reliability level using the Human Error Assessment and Reduction Technique (HEART) method. IOP Conference Series: Materials Science and Engineering, Vol. 830, 2020, pages 032095, doi:10.1088/1757- 899X/830/3/032095. [9] D. Safitri, R. Astriaty, N. Rizani, Human Reliability Assessment dengan Metode Human Error Assessment and Reduction Technique pada Operator Stasiun Shroud PT.X, Jurnal Rekayasa Sistem Industri Vol. 4, No. 1, (2015) 1–7. doi:10.26593/jrsi.v4i1.1388.1-7. [10] H. Ahmad, M. Faizal, M. Mazleha, Research reactor operator performance based on the human error assessment and reduction technique (HEART), in IOP Conference Series: Materials Science and Engineering, Vol. 785, International Nuclear Science Technology and Engineering Conference 2019 (iNuSTEC2019) 29–31 October 2019, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia. doi:10.1088/1757-899x/785/1/012048. [11] H. Che, S. Zeng, J. Guo, Reliability assessment of man-machine systems subject to mutually dependent machine degradation and human errors. Reliab Eng Syst, Vol. 190. (2019) 106504. doi:10.1016/j.ress.2019.106504. [12] Y. Fan, Study of a new Analysis Method of Risk Priority Number Based on FMEA, SAE Technical Paper 2015-01-0136. (2015) doi:10.4271/2015-01-0136. [13] W. Chappelle, T. Goodman, J. Swearingen, W. Thompson. A Preliminary Investigation into Cognitive Aptitudes Predictive of Overall MQ-1 Predator Pilot Qualification Training Performance. Wright-Patterson AFB (OH): U.S. Air Force School of Aerospace Medicine, 2015. Technical Report AFRL-SA-WP-SR-2015-0025. 14 p. [14] A. Noroozian; A. Sayyah, A new hybrid reliability allocation method, based on developed ME- OWA method, Birnbaum importance measure and fuzzy DEMATEL technique, International Journal of Critical Infrastructures (IJCIS), Vol. 16, No. 2. (2020) 162–185. doi:10.1504/IJCIS.2020.107269. [15] D. Kwon, A. Lucero, Optimized reliability test design to reduce uncertainties in reliability assessment in: 62nd. Electronic Components and Technology Conference, San Diego, CA, USA. (2012) 1726–1730, doi: 10.1109/ECTC.2012.6249070. [16] J. Lee, M. Mitici, An integrated assessment of safety and efficiency of aircraft maintenance strategies using agent-based modelling and stochastic Petri nets, Reliability Engineering & System Safety, Vol. 202, 107052 (2020) doi:10.1016/j.ress.2020.107052. [17] Werner L. Probability Theory. De Gruyter Textbook (2016) 395 p. doi:10.1515/9783110466195. [18] I. Korobiichuk, A. Tokar, Y. Danik, V. Katuha, in: Evaluation methods for the ergatic system reliability operator. Automation 2019: progress in automation, robotics and measurement techniques, Vol. 920 (2020) 560–570. doi:10.1007/978-3-030-13273-6_52. [19] L. Ying, G. Wen-xiang, A plan recognition algorithm based on the probabilistic goal graph, in: 2011 International Conference on Network Computing and Information Security, Guilin, 359– 362. doi:10.1109/ncis.2011.79. [20] Graph Theory, in: Probabilistic Networks and Expert Systems. Information Science and Statistics. Springer, New York, NY (1999) doi:10.1007/0-387-22630-3_4. [21] M. Bar Eli, H. Plessner, M. Raab, Theories of Decision Making, in Judgement, Decision Making and Success in Sport (2011) 27–38. doi: 10.1002/9781119977032.ch3. [22] J. Mishra, D. Allen, A. Pearman, Information seeking, use, and decision making, Journal of the Association for Information Science and Technology, vol. 66(4). (2015) 662–673. doi: 10.1002/asi.23204.