=Paper=
{{Paper
|id=Vol-2864/paper22
|storemode=property
|title=Methodology for Evaluation the Performance Indicators of the Ergatic Information System Functioning
|pdfUrl=https://ceur-ws.org/Vol-2864/paper22.pdf
|volume=Vol-2864
|authors=Tetiana Vakaliuk,Ihor Pilkevych,Andrii Tokar,Roman Loboda
|dblpUrl=https://dblp.org/rec/conf/cmis/VakaliukPTL21
}}
==Methodology for Evaluation the Performance Indicators of the Ergatic Information System Functioning==
https://ceur-ws.org/Vol-2864/paper22.pdf
Methodology for Evaluation the Performance Indicators of the
Ergatic Information System Functioning
Tetiana Vakaliuka, Ihor Pilkevychb, Andrii Tokarb and Roman Lobodab
a
Zhytomyr Polytechnic State University, Chudnivska str., 103, Zhitomir, 10005, Ukraine
b
Zhytomyr military institute named after S.P. Korolov, Mira ave., 22, Zhitomir, 10004, Ukraine
Abstract
The paper proposes a generalized methodology for evaluation the performance indicators of
the ergatiс information system, which is based on an algorithmic model of the operator's
activity and takes into account its reliability. For this purpose, the process of operator activity
is represented in the form probabilistic graph, which is a graphical representation of
individual operations and allows to describe the algorithm mathematically, to evaluate its
hourly and probabilistic indicators using the rules of transformation graph-scheme.
Proceeding from the purpose of the ergatiс information system, the tasks that it solves, the
indicators are selected the effectiveness of its operation. The order of their calculation is
given. The influence exerted by the operator reliability on the efficiency performance of the
ergatiс information system is taken into account by means a coefficient which characterizes
the operator's ability to work. By the example radio-monitoring system calculations
indicators efficiency of its functioning are carried out.
Keywords 1
Coefficient the reduction of operator reliability, ergatic system, operator reliability,
performance indicators, system functioning performance.
1. Introduction
The efficiency of functioning within any ergatiс information system depends on the reliability of
its components: technical means and operator. The modern science allows us to establish successfully
the laws of occurrence failure occurrence of devices and methods to forecast them, to find methods to
improve reliability during their design and the following manufacturing, as well as methods and
techniques to maintain reliability during their storage and operation. At the same time, the individual
nature and high variability of human psychological, physiological and professional capabilities and
characteristics, its sensitivity to the influence exerted by external and internal environment factors
complicates the processes related to analysis, forecasting and improvement of human-operator
reliability. As a consequence, due to human errors as a result of her insufficient training, unfavorable
psychological factors, fatigue occur the majority share in all the accidents and accidents in different
branches of activity [1-3]. Under such conditions, the issue relating to the evaluation indicators of the
effectiveness at functioning ergatiс information system is topical. Under such system we will
understand the interaction of technical means and the operator, whose activity is aimed at the timely
identification and processing significant volume of information flows, rapid analysis of the received
data, development an optimum decision concerning the further actions. Examples such activities are
the actions of an air traffic controller, operator unmanned aircraft complex, operator of a radio
monitoring post and the like.
CMIS-2021: The Fourth International Workshop on Computer Modeling and Intelligent Systems, April 27, 2021, Zaporizhzhia, Ukraine
EMAIL: tetianavakaliuk@gmail.com (T. Vakaliuk); igor.pilkevich@meta.ua (I. Pilkevych); tapir@i.ua (A. Tokar);
romaloboda0704@gmail.com (R. Loboda)
ORCID: 0000-0001-6825-4697 (T. Vakaliuk); 0000-0001-5064-3272 (I. Pilkevych); 0000-0001-7534-2820 (A. Tokar); 0000-0003-4010-
0252 (R. Loboda)
© 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
To prevent a failure in the execution of tasks, it is necessary to evaluate the performance indicators
of the information system in the process of its operation and compare their values with the limiting
value. Based on this, there is an urgent need to calculate these indicators in real time, taking into
account the peculiarities of the operator's activities in specific conditions, in order to prevent the
deterioration of the quality of tasks performed by the system as a whole. To do this, we will formalize
and simulate the process of the system functioning, as a result of which we will obtain the dependence
of the performance indicators of the information system on the operator's reliability. When
constructing a model, we will determine its qualitative composition and numerical parameters.
2. Review of the literature
Sufficiently well-developed approaches to the evaluation efficiency of systems at the stage of their
design [4, 5]. Estimation a degree of growth probability human error depending on changes of
environmental conditions using prediction model is described in [3], but its influence on the efficiency
the system is not shown. Impact that system reliability and task uncertainty have on unmanned aerial
vehicle operator performance during patrolling and target recognition tasks is investigated in [6], but
the inverse effect is not explored. The issue concerning quantitative estimation probability of human
error on the basis the qualitative analysis human factor and task context with additional consideration
of factors forming performance is devoted to the research work [7]. In research [8] measurement from
the operator's reliability level is carried out using Human Error Assessment and Reduction Technique
(HEART) on the example of 3 male operators using computer numerical control milling with work
experience > 3 months, operator age between 18 and 39 years. Method HEART is proposed to be
applied to analyze human realibility in production by is done using Hierarchical Task Analysis in
research [9]. In [10] the HEART method is used in order to quantify the human failure event for
probabilistic safety assessment of a TRIGA Mark II reactor, situated at the Malaysia Nuclear Agency,
Malaysia. This approach requires the selection values nominal probabilities of human error and does
not take into account the time factor. A mutual dependence between machine degradation and human
error in man-machine systems are studied in research [11], but the influence of other factors on the
efficiency of the system functioning is not taken into account. In [12], it is proposed to evaluate the
efficiency of the system using some quantitative parameters such as servicing time and costs. This
approach does not take into account the degree of decrease in the reliability of the system user. The
issue of professional suitability at the stage of selection of candidates for training pilots of the United
States Air Force MQ-1 Predator is investigated in [13]. The study of changes in the efficiency of the
MQ-1 Predator depending on the reliability of the operator in this work is not performed. A number of
works are devoted to assessing and improving the reliability and efficiency of systems functioning,
but at the same time, no attention is paid to the influence of operator reliability on the performance
indicators of the system as a whole [14-16].
Analysis shows that quite successfully solved the issue concerning the evaluation of the
effectiveness systems at the stage their design. At the same time, during the assessment and
improvement of the reliability and efficiency of the systems functioning, attention is not paid to the
influence of the operator's reliability on the performance of the system as a whole. In the direction of
evaluation efficiency systems in their functioning studies conducted limited to the assessment a
person's reliability without calculating the performance measures of the system as a whole, in addition
is not taken into account the factor changes in the probability or human errors over time.
So, the purpose the work is to develop a methodology for evaluation the performance indicators of
the ergatiс information system, taking into account the reliability of the operator.
3. Research methods
The functioning of the ergatic information system will be considered as a process of timely
identification, receipt and processing of information flows by the operator, rapid analysis of the data
obtained, the development of optimal solutions for further action. In this case, we will assume that the
technical means of the information system work faultlessly at a given time interval. Under such
conditions, the effectiveness of the functioning of the ergatiс information system will be determined
by the timeliness and validity of decision-making by the operator. The probability of a certain event is
used as performance indicators in conditions of uncertainty [3, 6–8]. Therefore, as indicators of
efficiency of functioning of ergatic information system we use probability of timely decision making
– Pt and probability of error-free decision making – Pd by operator of information system.
3.1. Method for evaluation the probability of timely decision-making
The probability of timely decision-making by the operator of information system – Pt
characterizes the probability of event Tmin Po Ta , that is
P P(T P T ), (1)
t min o a
where Tmin – is the minimum possible time of the operator's decision; Ta – is the maximum allowable
time of the operator's decision; To – time is the operator's decision.
According to probability theory [17]
P(Tmin Po Ta ) F(Ta ) F(Tmin ) , (2)
where F(Ta ) – is the time distribution function Ta ; F(Tmin ) – is the time distribution function Tmin .
Due to the influence of a large number and random variables on the information system operator
(increase or redistribution efforts from information processing bodies when the situation changes;
constant influence on the operator the results from the information flows processing; failures in the
regulation process and reporting the data received, decrease in personnel training, deterioration of
communication channels quality, quality level of information base used; lack as well as inadequate
resources to implement processing in case of complicated situation on the information system
operator's side) there is every reason to believe that the time for making a decision by the operator is
distributed according to the normal law [17].
In such a case the density of the distribution function of the operator's decision-making time will
be equal to
( t m ) 2
1 2 2 (3)
f (t ) e
2 .
From expression (3) we obtain the time distribution functions Tmin and Ta
( Tmin m )2
1 Tmin 2 2 (4)
F ( Tmin ) e dTmin ,
2
( Ta m )2
1 Ta 2 2 (5)
F ( Ta ) e dTa ,
2
where – is the standard deviation of the operator's decision time; m – is the mathematical
expectation of the operator's decision time.
Expressing the distribution functions (4) and (5) through the standard distribution function (6)
( x ) 2
1 x 2 (6)
Ф( x ) e dx ,
2
using expressions (1) and (2) we obtain
T m T m
Pt P(Tmin Po Ta ) Ф( a ) Ф( min ). (7)
Over time, as a result for various reasons, when an operator makes a decision, and hence the
mathematical expectation time of making a decision will increase and is determined by the
dependence m m t , where t – is the value of the increase in the average time of the operator's
decision due to a decrease in its reliability
t ( Tmax m)K R , (8)
where K R – is a coefficient characterizing the reduction of operator reliability. In the optimal state
operator – K R 0 , in a critical decrease in the reliability operators K R 1 . Procedure for calculation
of K R is given in [18]; Tmax – time for decision making at a low level of reliability the operator of
the information system, i.e. when K R 1 .
Taking into account (7), (8), we obtain an expression for the evaluation of the probability of timely
decision making by the information system operator at any point of time
T ( m t ) T ( m t
Pt Ф( a ) Ф( min )
(9)
Ta Tmax K R m( 1 K R ) Tmin Tmax K R m( 1 K R
Ф( ) Ф( ).
To calculate the probability of timely decision-making by the operator at an arbitrary moment
using expression (9), it is necessary to know the minimum possible decision-making time – Tmin , the
maximum possible decision-making time by the operator – Ta , average decision-making time by the
operator in optimal conditions – m , decision-making time at a low level operator reliability – Tmax .
For definition of mentioned parameters, we will use method on the base an algorithm probability
graph [19, 20], which is based on analysis of algorithm of operator's activity. For the effective usage
algorithmic models, it is necessary to obtain reliable data on the elements of operator activity, i.e. to
analyze the functioning process in a particular information system. As a result, to establish the time,
probabilistic characteristics and accuracy indicators of the information system operator's activity,
based on the data given in the reference literature and their clarification, taking into account various
factors (operator's state, ergonomic characteristics of the system etc.).
For example, let us calculate the probability of a timely decision by the operator of the radio
monitoring system. For this, an analysis of the functioning of such a system based on a radio receiver
was carried out. As a result, based on the list of operations and logical conditions that were obtained
by the expert method, a structural diagram of the activity of the operator of the radio monitoring
information system was drawn up (Figure 1). A list of operations of the operator's activity algorithm
has been compiled and the time indicators of each operation are given – Tmini , Ta i , mi , Tmaxi
(Table 1).
Understanding Deciding on the Receiving radio Registration of data
the task operating mode monitoring data on tangible media
Accounting for the
Primary study Analysis Generalization
received data
Report
Figure 1: Block diagram of the activity of the operator of the radio monitoring information system
Table 1
Operator activity operations and their timing
№ Tmini , c Ta i , c Tmaxi , c
Operation name mi , c
operation
1 Understanding the task 25 50 30 60
2 Deciding on the operating mode 1,2 3 1,8 5
3 Switching the receiver to radio monitoring
mode 5 10 7 15
4 Switching the receiver into search mode 5 10 7 15
5 Signal processing decision making 1,5 3 2 4
6 Selection of the signal of interest 3 6 4 8
7 Pre‐latching (signal acquisition) 5 10 7 12
8 Preliminary analysis of the information
message 10 25 15 32
9 Determination of signal novelty 5 25 10 32
10 Sending a direction finding command 2,3 4,3 2,8 6
11 Getting results direction finding 2 5 3 7
12 Assignment of bearing to a specific source 5 10 7 16
13 Establishing the main features of the source 60 120 80 130
14 Source identification 1,5 3 2 5
15 Registration of data on tangible media 50 110 70 120
16 Familiarization with data content 60 160 85 174
17 Highlighting known and unknown elements 10 30 17 50
18 Determining the importance of a new
element 3 7 4 11
19 Determination of urgency 3 7 4 11
20 Report 10 20 11 30
21 Validity informations assessment 30 70 40 90
22 Determining the informations characteristics 40 80 50 110
23 Comprehensive data exploration, comparison
with available data 120 240 150 280
24 Identifying changes in condition and
character 5 15 8 17
25 Formation of conclusions 130 270 170 340
26 Formation of information documents 180 400 220 420
27 Thematic systematization 25 45 30 70
28 Accounting for the received data 120 250 170 310
29 Results report 10 20 11 30
Time indices Tmin , Ta , m , Tmax we will receive on the basis of time indices individual
operations Tmini , Ta i , mi , Tmaxi (Table 1) by representing the algorithm and using the rules of its
transformation [19, 20].
Probabilistic oriented graph, corresponding to the algorithm of radio monitoring system
functioning is shown in Figure 2. It is a graphical representation of individual operations and allows to
describe the algorithm mathematically, to estimate the time and probabilistic indicators.
Each vertex of the graph corresponds to an operation according to the numbers from Table 1 and a
4-dimensional vector Vi , whose components are Tmini , Ta i , mi , Tmaxi . The set graph vertices and
its branches form the graph-scheme of the algorithm. To obtain time indicators in the algorithm we
use the rules of graph-scheme transformation [20].
3 5
1 2 7 8 9 10 11 12
4 6
13 14 15 16
29 28 27 26 25 24 23 17
22
21 19 18
20
Figure 2: Probabilistic directed graph
1. Combining paths without branches.
The path graph without branches has the form (Figure 3):
1 2 …. n
V1 V2 Vn
Figure 3: Graph without branches
The concatenation operation turns a branch without branches into an equivalent branch with one
vertex (Figure 4).
1-n
Ve
Figure 4: Equivalent branch with one vertex
The components of the vector Ve are found by the expressions:
n
Tmin e Tmini
i 1
n
Ta e Tai
i 1
(10)
n
me mi
i 1
n
Tmax e Tmaxi
i 1
2. Combining paths with a branching.
The graph of a path with a branching has the form (Figure 5):
p 2
V2
1 4
V1 V4
1-p 3
V3
Figure 5: Graph of a path with a branching
The concurrent paths joining operation turns a branch with a branching into an equivalent branch
with one vertex with vector Ve whose components are found by the expressions:
Tmin e Tmin1 pTmin2 ( 1 p )Tmin3 Tmin4
;
Ta e Ta1 pTa2 ( 1 p )Ta3 Ta4
; (11)
me m1 pm2 ( 1 p )m3 m4 ;
Tmax e Tmax1 pTmax2 ( 1 p )Tmax3 Tmax4
,
where p – is the probability of transition along a branch, taken as 0,5 for all branches.
According to the given rules let us join paths without branches into graph (see Figure. 2) (1, 2), (3,
5), (4, 6), (7, 8, 9), (10, 11, 12 ), (13, 14, 15, 16, 17), (18, 19), (21, 22), (23, 24), (25, 26, 27, 28, 29).
As the result we obtain a probabilistic oriented graph presented in Figure 6.
18-19 20
10-12
3-5
1-2 7-9 13-17 21-22
4-6
23-24 25-29
Figure 6: Result of consecutive branch merging
Using expressions (10), we obtain equivalent time characteristics of graph vertices (Table 2).
Table 2
Time characteristics after combining
№ operation Tmini , c Ta i , c mi , c Tmaxi , c
1‐2 26,2 53 31,8 65
3‐5 6,5 13 9 19
4‐6 8 16 11 23
7‐9 20 60 32 76
10‐12 9,3 19,3 12,8 29
13‐17 131,5 313 184 359
18‐19 6 14 8 22
20 10 20 11 30
21‐22 70 150 90 200
23‐24 125 255 158 297
25‐29 465 925 601 1170
To combine paths with branching, accept the same probability to go through each branch of the
logical operator is p 0 ,5 . As a result combining paths with branching (1-2, 3-5, 4-6, 7-9) and (18-
19, 20, 21-22) the graph-scheme will look like (Figure 7).
10-12
18-22
1-9 13-17 25-29
23-24
Figure 7: Result of consecutive branch merging
Using expressions (11), we obtain equivalent time characteristics of graph vertices (Table 3).
Table 3
Time characteristics after combining
№ operation Tmini , c Ta i , c mi , c Tmaxi , c
1‐9 53,3 127,2 73,6 161,6
10‐12 9,3 19,3 12,8 29
13‐17 131,5 313 184 359
18‐22 81 174 103,5 237
23‐24 125 255 158 297
25‐29 465 925 601 1170
Combine the paths with branching (1-9, 10-12, 13-17) (Figure 8) and obtain the equivalent time
characteristics of the vertices of the graph (Table 4).
18-22
1-17 25-29
23-24
Figure 8: Result of consecutive branch merging
Table 4
Time characteristics after combining
№ operation Tmini , c Ta i , c mi , c Tmaxi , c
1‐17 187,59 446 261,4 529,3
18‐22 81 174 103,5 237
23‐24 125 255 158 297
25‐29 465 925 601 1170
After combining paths with branching (1-17, 18-22, 23-24, 25-29) (Figure 9), we obtain equivalent
time characteristics of the equivalent graph vertices (Table 5).
1-29
Figure 9: Result of consecutive branch merging
Table 5
Time characteristics after combining
№ operation Tmini , c Ta i , c mi , c Tmaxi , c
1‐29 818 1626 1072 1996
Obtained time indices algorithm activity of the radio monitoring system operator gives the
opportunity to find the probabilities of timely decision making by the operator at any point in time by
expression (9). Taking into account that in solving the problem the operator has normal law
distribution, the average deviation in time of taking the decision by the operator will be found by
expression
m Tmin
3 . (12)
The obtained value of the probability of a timely decision-making by the operator must be
compared with the admissible value. The permissible value can be obtained in further research in
relation to a specific task performed by the system.
So, the method for evaluation the probability of timely decision-making by the information system
operator will consist of the following steps:
1. Make a probabilistic oriented graph of the functioning of the ergatiс information system.
2. Determine the timing of individual operations – Tmini , Ta i , mi , Tmaxi .
3. Perform transformations of the graph using the rules of combining paths without branches and
with branches and calculate the time indices – Tmin , Ta , m , Tmax .
4. Calculate the average deviation of the operator's decision time by expression (12).
5. Calculate the probability of timely decision-making by the operator of the information system
by expression (9).
3.2. Method for evaluation the probability of error-free decision-making
To calculate the probability of error-free decision making by an information system operator we
will use a method based on information analysis of decision making processes [21, 22]. The basis of
the informational approach is the fact that the most essential factor, which influences the error-free
decision making, is the volume of initial information. Accordingly, for any complex system, an
increase in the amount of initial information leads to an increase in the probability of error-free
decision making by an operator. Taking into account loss of initial information due to decrease of
operator's reliability, let us write the expression for probability of error-free decision – Pd of the i -th
operation:
( I in I l )
Ki (13)
Pd i 1 Y0 e ,
where I in – amount of initial information; I l – amount of information lost; Y0 – initial uncertainty
of the decision taken characterizes the psychological confidence of the operator (at the optimal state
of the operator Y0 1 ); K i – coefficient that characterizes the value an information and affects the
growth rate of probability depending on the amount of processed information by the operator. At
maximum value of the information K i tends to 0, at minimum – K i = 1. The value of this coefficient
will be found on the assumption that as the number of possible states of the system, and consequently
the number of variants of decisions n , grows, the value of the information by the expression
1
Ki . (14)
n
The amount of initial information about some system is equal to the entropy of this system
I in N , and at equal-probability n states of the system [17]
I in log n. (15)
Over time, due to the impact of various reasons, part the original information will be lost, the value
of the loss will depend on the level efficiency of the operator and determined by the introduced
coefficient decrease in the reliability of the operator – K R . The amount is lost information we will
find by the expression.
I l I in .K R . (16)
The level of reliability will also affect the level in psychological confidence, that is, the coefficient
Y0 will decrease in accordance with the value of K R .
With the decrease of the level a reliability operators value of information will decrease. To take
this into account let's write expression (14) in the form:
K
Ki R . (17)
n
Given expressions (13), (15), (16), (17), and assuming that the operator solution is completely
indeterminate is Y0 =1, we obtain an expression for determining the probability of error-free solution
at the i -th operation at an arbitrary time moment:
n ( 1 K R ) log ni
i
KR (18)
Pd i 1 K R e .
To carry out calculations on the example the radio monitoring system, by expert polling was
established the number from the i -th operation, determining the number of states – n (Table 6,
Table 7). In the table the numbers of operations correspond to the numbers from Table 1.
Table 6
Time characteristics after combining
№ operation 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Number of
3 2 2 2 2 4 4 13 4 4 4 4 14 7 9
possible solutions
Table 7
Time characteristics after combining
№ operation 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Number of possible
9 16 16 16 2 16 10 9 16 2 2 4 5 2
solutions
To obtain the probability for error-free decision by the operator of the radio-monitoring system,
was used probabilistic directed graph, represented on Figure 2. In the process of transforming the
initial graph-scheme to the equivalent one, with one vertex, the union of the same paths was carried
out, as in determining the time characteristics, taking into account the rules for transforming the graph
[20]. It was taken into account that when merging paths without branching (Figure 10)
1 2 …. n
Pd1 Pd2 Pdn
Figure 10: Paths without branching
probability characteristics are determined by the expression [20]
n
Pd Pd i . (19)
i 1
When combining tracks with branching (Figure 11)
p 2 P2
P1 P4
1 4
P3
1-p 3
Figure 11: Paths without branching
probability characteristics are determined by the expression
Pd P1P4 ( pP2 ( 1 p )P3 ). (20)
So, the method for evaluation the probability of error-free decision-making by the information
system operator will consist of the following steps:
1. Make a probabilistic oriented graph for functioning of an ergatiс information system.
2. Determine the number or possible solutions of the i -th operation determining the number of
states – n .
3. Calculate the probability for error-free decision by the operator when performing the i -th
operation at an arbitrary moment of time by expression (18).
4. To carry out transformations to the graph using the rules of combining paths without branching
and with branching and to carry out calculation – Pd by expressions (19), (20).
In order to verify the proposed methods for evaluation the performance of the ergatic information
system, the probabilities involved in the timely decision-making and the probability of error-free
decision-making by the operator information system for the range of changes in the coefficient
K R 0 1 were calculated. The results are shown in Table 8, for which the corresponding
dependencies were built (Figure 12).
Table 8
Calculation results
KR Pt Pd
0 1,00 1,00
0,1 0,99 0,99
0,2 0,99 0,99
0,3 0,99 0,99
0,4 0,94 0,97
0,5 0,81 0,85
0,6 0,58 0,50
0,7 0,28 0,13
0,8 0,05 0,02
0,9 0,01 0,01
1 0,00 0,00
Pd , Pt
Pd
Pt
KR
Figure 12: Dependence of indicators on K R
Analysis of the data obtained shows that at the initial stage of reducing the operator's reliability (up
to K R = 0.4), the performance indicators of the system functioning practically do not change. This
can be explained by the inclusion of human compensatory mechanisms. After a certain value, a sharp
decrease in the performance indicators of the system functioning to critical values is observed.
4. Conclusions
The developed methods constitute the content of the generalized methodology for evaluation the
performance indicators for ergatic information system, based on an algorithmic model of the
operator's activity. The methods take into account the reduction factor of operator reliability, made it
possible to obtain an estimate of probabilities for timely decision-making and probability of error-free
decision-making by the information system operator in real time while on duty.
5. References
[1] T. Kletz, P. Amyotte, Accidents said to be due to human error, What Went Wrong?, 6th. ed.
Butterworth-Heinemann. (2019) 53–172. doi:10.1016/B978-0-12-810539-9.00007-0.
[2] B. Strauch, Investigating Human Error: Incidents, Accidents, and Complex Systems, 1st. ed.,
Routledge (2018) doi:10.4324/9781315251851.
[3] Y. Tan, D. Feng, H. Shen, Research for Unmanned Aerial Vehicle components reliability
evaluation model considering the influences of human factors, 3rd International Conference on
Mechanical, Electronic and Information Technology Engineering., ICMITE 2017, vol. 139,
article number 00221. doi:10.1051/matecconf/201713900221.
[4] Z. Mian, D. Mavris, Mission effectiveness quantification and assessment of Micro Autonomous
Systems and Technologies, IEEE Systems and Information Engineering Design Symposium
(2013) doi:10.1109/SIEDS.2013.6549514.
[5] Haifeng Zhu, Fundamental Models for Missions of Engineered Systems, Systems Conference
(SysCon) IEEE International. (2019) 1–6. doi:10.1109/syscon.2019.8836714.
[6] D. Liu, M. Jaramillo, D. Vincenzi, The effects of system reliability and task uncertainty on
autonomous unmanned aerial vehicle operator performance under high time pressure. Hum.
Factors Ergon. Vol. 25, № 5. (2015) 515–522. doi:10.1002/hfm.20565.
[7] X. Pan, Z. Wu, Performance shaping factors in the human error probability modification of
human reliability analysis, International Journal of Occupational Safety and Ergonomics,
Vol. 26. (2020) 538–550, doi:10.1080/10803548.2018.1498655.
[8] Y. Rejeki, E. Achiraeniwati, A. Wanda, Measurement of operator reliability level using the
Human Error Assessment and Reduction Technique (HEART) method. IOP Conference Series:
Materials Science and Engineering, Vol. 830, 2020, pages 032095, doi:10.1088/1757-
899X/830/3/032095.
[9] D. Safitri, R. Astriaty, N. Rizani, Human Reliability Assessment dengan Metode Human Error
Assessment and Reduction Technique pada Operator Stasiun Shroud PT.X, Jurnal Rekayasa
Sistem Industri Vol. 4, No. 1, (2015) 1–7. doi:10.26593/jrsi.v4i1.1388.1-7.
[10] H. Ahmad, M. Faizal, M. Mazleha, Research reactor operator performance based on the human
error assessment and reduction technique (HEART), in IOP Conference Series: Materials
Science and Engineering, Vol. 785, International Nuclear Science Technology and Engineering
Conference 2019 (iNuSTEC2019) 29–31 October 2019, Universiti Kebangsaan Malaysia, 43600
Bangi, Selangor, Malaysia. doi:10.1088/1757-899x/785/1/012048.
[11] H. Che, S. Zeng, J. Guo, Reliability assessment of man-machine systems subject to mutually
dependent machine degradation and human errors. Reliab Eng Syst, Vol. 190. (2019) 106504.
doi:10.1016/j.ress.2019.106504.
[12] Y. Fan, Study of a new Analysis Method of Risk Priority Number Based on FMEA, SAE
Technical Paper 2015-01-0136. (2015) doi:10.4271/2015-01-0136.
[13] W. Chappelle, T. Goodman, J. Swearingen, W. Thompson. A Preliminary Investigation into
Cognitive Aptitudes Predictive of Overall MQ-1 Predator Pilot Qualification Training
Performance. Wright-Patterson AFB (OH): U.S. Air Force School of Aerospace Medicine, 2015.
Technical Report AFRL-SA-WP-SR-2015-0025. 14 p.
[14] A. Noroozian; A. Sayyah, A new hybrid reliability allocation method, based on developed ME-
OWA method, Birnbaum importance measure and fuzzy DEMATEL technique, International
Journal of Critical Infrastructures (IJCIS), Vol. 16, No. 2. (2020) 162–185.
doi:10.1504/IJCIS.2020.107269.
[15] D. Kwon, A. Lucero, Optimized reliability test design to reduce uncertainties in reliability
assessment in: 62nd. Electronic Components and Technology Conference, San Diego, CA, USA.
(2012) 1726–1730, doi: 10.1109/ECTC.2012.6249070.
[16] J. Lee, M. Mitici, An integrated assessment of safety and efficiency of aircraft maintenance
strategies using agent-based modelling and stochastic Petri nets, Reliability Engineering &
System Safety, Vol. 202, 107052 (2020) doi:10.1016/j.ress.2020.107052.
[17] Werner L. Probability Theory. De Gruyter Textbook (2016) 395 p. doi:10.1515/9783110466195.
[18] I. Korobiichuk, A. Tokar, Y. Danik, V. Katuha, in: Evaluation methods for the ergatic system
reliability operator. Automation 2019: progress in automation, robotics and measurement
techniques, Vol. 920 (2020) 560–570. doi:10.1007/978-3-030-13273-6_52.
[19] L. Ying, G. Wen-xiang, A plan recognition algorithm based on the probabilistic goal graph, in:
2011 International Conference on Network Computing and Information Security, Guilin, 359–
362. doi:10.1109/ncis.2011.79.
[20] Graph Theory, in: Probabilistic Networks and Expert Systems. Information Science and
Statistics. Springer, New York, NY (1999) doi:10.1007/0-387-22630-3_4.
[21] M. Bar Eli, H. Plessner, M. Raab, Theories of Decision Making, in Judgement, Decision Making
and Success in Sport (2011) 27–38. doi: 10.1002/9781119977032.ch3.
[22] J. Mishra, D. Allen, A. Pearman, Information seeking, use, and decision making, Journal of the
Association for Information Science and Technology, vol. 66(4). (2015) 662–673. doi:
10.1002/asi.23204.