Methodology for Evaluation the Performance Indicators of the Ergatic Information System Functioning

Abstract

   The paper proposes a generalized methodology for evaluation the performance indicators of the ergatiс information system, which is based on an algorithmic model of the operator's activity and takes into account its reliability. For this purpose, the process of operator activity is represented in the form probabilistic graph, which is a graphical representation of individual operations and allows to describe the algorithm mathematically, to evaluate its hourly and probabilistic indicators using the rules of transformation graph-scheme. Proceeding from the purpose of the ergatiс information system, the tasks that it solves, the indicators are selected the effectiveness of its operation. The order of their calculation is given. The influence exerted by the operator reliability on the efficiency performance of the ergatiс information system is taken into account by means a coefficient which characterizes the operator's ability to work. By the example radio-monitoring system calculations indicators efficiency of its functioning are carried out.  

1. Introduction 

The efficiency of functioning within any ergatiс information system depends on the reliability of its components: technical means and operator. The modern science allows us to establish successfully the laws of occurrence failure occurrence of devices and methods to forecast them, to find methods to improve reliability during their design and the following manufacturing, as well as methods and techniques to maintain reliability during their storage and operation. At the same time, the individual nature and high variability of human psychological, physiological and professional capabilities and characteristics, its sensitivity to the influence exerted by external and internal environment factors complicates the processes related to analysis, forecasting and improvement of human-operator reliability. As a consequence, due to human errors as a result of her insufficient training, unfavorable psychological factors, fatigue occur the majority share in all the accidents and accidents in different branches of activity [1-3]. Under such conditions, the issue relating to the evaluation indicators of the effectiveness at functioning ergatiс information system is topical. Under such system we will understand the interaction of technical means and the operator, whose activity is aimed at the timely identification and processing significant volume of information flows, rapid analysis of the received data, development an optimum decision concerning the further actions. Examples such activities are the actions of an air traffic controller, operator unmanned aircraft complex, operator of a radio monitoring post and the like.

To prevent a failure in the execution of tasks, it is necessary to evaluate the performance indicators of the information system in the process of its operation and compare their values with the limiting value. Based on this, there is an urgent need to calculate these indicators in real time, taking into account the peculiarities of the operator's activities in specific conditions, in order to prevent the deterioration of the quality of tasks performed by the system as a whole. To do this, we will formalize and simulate the process of the system functioning, as a result of which we will obtain the dependence of the performance indicators of the information system on the operator's reliability. When constructing a model, we will determine its qualitative composition and numerical parameters.

2. Review of the literature 

Sufficiently well-developed approaches to the evaluation efficiency of systems at the stage of their design [4, 5]. Estimation a degree of growth probability human error depending on changes of environmental conditions using prediction model is described in [3], but its influence on the efficiency the system is not shown. Impact that system reliability and task uncertainty have on unmanned aerial vehicle operator performance during patrolling and target recognition tasks is investigated in [6], but the inverse effect is not explored. The issue concerning quantitative estimation probability of human error on the basis the qualitative analysis human factor and task context with additional consideration of factors forming performance is devoted to the research work [7]. In research [8] measurement from the operator's reliability level is carried out using Human Error Assessment and Reduction Technique (HEART) on the example of 3 male operators using computer numerical control milling with work experience > 3 months, operator age between 18 and 39 years. Method HEART is proposed to be applied to analyze human realibility in production by is done using Hierarchical Task Analysis in research [9]. In [10] the HEART method is used in order to quantify the human failure event for probabilistic safety assessment of a TRIGA Mark II reactor, situated at the Malaysia Nuclear Agency, Malaysia. This approach requires the selection values nominal probabilities of human error and does not take into account the time factor. A mutual dependence between machine degradation and human error in man-machine systems are studied in research [11], but the influence of other factors on the efficiency of the system functioning is not taken into account. In [12], it is proposed to evaluate the efficiency of the system using some quantitative parameters such as servicing time and costs. This approach does not take into account the degree of decrease in the reliability of the system user. The issue of professional suitability at the stage of selection of candidates for training pilots of the United States Air Force MQ-1 Predator is investigated in [13]. The study of changes in the efficiency of the MQ-1 Predator depending on the reliability of the operator in this work is not performed. A number of works are devoted to assessing and improving the reliability and efficiency of systems functioning, but at the same time, no attention is paid to the influence of operator reliability on the performance indicators of the system as a whole [14-16].

Analysis shows that quite successfully solved the issue concerning the evaluation of the effectiveness systems at the stage their design. At the same time, during the assessment and improvement of the reliability and efficiency of the systems functioning, attention is not paid to the influence of the operator's reliability on the performance of the system as a whole. In the direction of evaluation efficiency systems in their functioning studies conducted limited to the assessment a person's reliability without calculating the performance measures of the system as a whole, in addition is not taken into account the factor changes in the probability or human errors over time.

So, the purpose the work is to develop a methodology for evaluation the performance indicators of the ergatiс information system, taking into account the reliability of the operator.

3. Research methods 

The functioning of the ergatic information system will be considered as a process of timely identification, receipt and processing of information flows by the operator, rapid analysis of the data obtained, the development of optimal solutions for further action. In this case, we will assume that the technical means of the information system work faultlessly at a given time interval. Under such conditions, the effectiveness of the functioning of the ergatiс information system will be determined by the timeliness and validity of decision-making by the operator. The probability of a certain event is used as performance indicators in conditions of uncertainty [3, 6–8]. Therefore, as indicators of efficiency of functioning of ergatic information system we use probability of timely decision making – Pt and probability of error-free decision making – Pd by operator of information system.  3.1.

Method for evaluation the probability of timely decision-making 

The probability of timely decision-making by the operator of information system –

P t characterizes the probability of event Tmin  Po  Ta , that is

Pt  P(Tmin  Po  Ta ) , where Tmin – is the minimum possible time of the operator's decision; Ta – is the maximum allowable time of the operator's decision; To – time is the operator's decision.

According to probability theory [17]

P(Tmin  Po  Ta )  F(Ta )  F(Tmin ) , where F(Ta ) – is the time distribution function Ta ; F(Tmin ) – is the time distribution function Tmin .

Due to the influence of a large number and random variables on the information system operator (increase or redistribution efforts from information processing bodies when the situation changes; constant influence on the operator the results from the information flows processing; failures in the regulation process and reporting the data received, decrease in personnel training, deterioration of communication channels quality, quality level of information base used; lack as well as inadequate resources to implement processing in case of complicated situation on the information system operator's side) there is every reason to believe that the time for making a decision by the operator is distributed according to the normal law [17].

In such a case the density of the distribution function of the operator's decision-making time will be equal to

From expression (3) we obtain the time distribution functions Tmin and Ta

f ( t )   1 2 F( Tmin ) 

F( Ta )    1 2

Tmin

 e  1

Ta  e 2  e ( t m )2 2 2

. ( Tmin m )2

2 2 ( Ta m )2 using expressions ( 1 ) and ( 2 ) we obtain

Pt  P(Tmin  Po  Ta )  Ф( Ta  m )  Ф( Tmin  m ) .

  t  ( Tmax  m)K R ,

Over time, as a result for various reasons, when an operator makes a decision, and hence the mathematical expectation time of making a decision will increase and is determined by the dependence m  m  t , where t – is the value of the increase in the average time of the operator's decision due to a decrease in its reliability where K R – is a coefficient characterizing the reduction of operator reliability. In the optimal state operator – K R  0 , in a critical decrease in the reliability operators K R  1 . Procedure for calculation of K R is given in [18]; Tmax – time for decision making at a low level of reliability the operator of the information system, i.e. when K R  1 .

Taking into account ( 7 ), ( 8 ), we obtain an expression for the evaluation of the probability of timely decision making by the information system operator at any point of time

Pt  Ф( Ta  ( m  t ) )  Ф( Tmin  ( m  t ) 

   Ф( Ta  Tmax K R  m( 1  K R ) )  Ф( Tmin  Tmax K R  m( 1  K R ).

 

To calculate the probability of timely decision-making by the operator at an arbitrary moment using expression ( 9 ), it is necessary to know the minimum possible decision-making time – Tmin , the maximum possible decision-making time by the operator – Ta , average decision-making time by the operator in optimal conditions – m , decision-making time at a low level operator reliability – Tmax .

For definition of mentioned parameters, we will use method on the base an algorithm probability graph [19, 20], which is based on analysis of algorithm of operator's activity. For the effective usage algorithmic models, it is necessary to obtain reliable data on the elements of operator activity, i.e. to analyze the functioning process in a particular information system. As a result, to establish the time, probabilistic characteristics and accuracy indicators of the information system operator's activity, based on the data given in the reference literature and their clarification, taking into account various factors (operator's state, ergonomic characteristics of the system etc.).

For example, let us calculate the probability of a timely decision by the operator of the radio monitoring system. For this, an analysis of the functioning of such a system based on a radio receiver was carried out. As a result, based on the list of operations and logical conditions that were obtained by the expert method, a structural diagram of the activity of the operator of the radio monitoring information system was drawn up (Figure 1). A list of operations of the operator's activity algorithm has been compiled and the time indicators of each operation are given – Tmini , Tai , mi , Tmaxi (Table 1).

Understanding the task

Deciding on the

operating mode

Receiving radio

monitoring data

Registration of data on tangible media

( 7 )  ( 8 )  ( 9 ) 

Primary study Analysis Generalization

Accounting for the received data

Report

Figure 1: Block diagram of the activity of the operator of the radio monitoring information system   4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29 

Understanding the task  Deciding on the operating mode  Switching the receiver to radio monitoring  mode  Switching the receiver into search mode  Signal processing decision making  Selection of the signal of interest  Pre‐latching (signal acquisition)  Preliminary analysis of the information  message  Determination of signal novelty  Sending a direction finding command  Getting results direction finding  Assignment of bearing to a specific source  Establishing the main features of the source  Source identification  Registration of data on tangible media  Familiarization with data content  Highlighting known and unknown elements  Determining the importance of a new  element  Determination of urgency  Report  Validity informations assessment   Determining the informations characteristics  Comprehensive data exploration, comparison  with available data  Identifying changes in condition and  character  Formation of conclusions  Formation of information documents  Thematic systematization  Accounting for the received data   Results report 

Time indices Tmin ,  Ta ,  m ,  Tmax   we will receive on the basis of time indices individual operations Tmini , Tai , mi , Tmaxi (Table 1) by representing the algorithm and using the rules of its transformation [19, 20].

Probabilistic oriented graph, corresponding to the algorithm of radio monitoring system functioning is shown in Figure 2. It is a graphical representation of individual operations and allows to describe the algorithm mathematically, to estimate the time and probabilistic indicators.

Each vertex of the graph corresponds to an operation according to the numbers from Table 1 and a 4-dimensional vector Vi , whose components are Tmini , Tai , mi , Tmaxi . The set graph vertices and its branches form the graph-scheme of the algorithm. To obtain time indicators in the algorithm we use the rules of graph-scheme transformation [20]. 50  3  10  10  3  6  10 

1. Combining paths without branches.

The path graph without branches has the form (Figure 3):

The concatenation operation turns a branch without branches into an equivalent branch with one vertex (Figure 4).

The components of the vector Ve are found by the expressions: ( 10 ) 

2. Combining paths with a branching.

The graph of a path with a branching has the form (Figure 5):

n Tmine   Tmini i1 n Tae  i1Tai

n me   mi i1 n Tmaxe   Tmaxi

i1 1 V1 p 1-p 2 V2 3 V3 Ve 25 22 21 ….

20 4 V4 24 19 n Vn

The concurrent paths joining operation turns a branch with a branching into an equivalent branch with one vertex with vector Ve whose components are found by the expressions:

Tmine  Tmin1  pTmin2  ( 1  p )Tmin3  Tmin4 ;

Tae  Ta1  pTa2  ( 1  p )Ta3  Ta4 ;

me  m1  pm2  ( 1  p )m3  m4 ;

Tmaxe  Tmax1  pTmax2  ( 1  p )Tmax3  Tmax4 , where p – is the probability of transition along a branch, taken as 0,5 for all branches.

According to the given rules let us join paths without branches into graph (see Figure. 2) ( 1, 2 ), ( 3, 5 ), ( 4, 6 ), ( 7, 8, 9 ), ( 10, 11, 12 ), ( 13, 14, 15, 16, 17 ), (18, 19), (21, 22), (23, 24), (25, 26, 27, 28, 29). As the result we obtain a probabilistic oriented graph presented in Figure 6.

( 11 )  10-12 1-2 7-9

13-17

Using expressions (10), we obtain equivalent time characteristics of graph vertices (Table 2).

To combine paths with branching, accept the same probability to go through each branch of the logical operator is p  0,5 . As a result combining paths with branching ( 1-2, 3-5, 4-6, 7-9 ) and (1819, 20, 21-22) the graph-scheme will look like (Figure 7).

Figure 7: Result of consecutive branch merging 

10-12 1-9 13-17 18-22 23-24 25-29  

Using expressions (11), we obtain equivalent time characteristics of graph vertices (Table 3).

Table 3  Time characteristics after combining 

Combine the paths with branching ( 1-9, 10-12, 13-17 ) (Figure 8) and obtain the equivalent time characteristics of the vertices of the graph (Table 4).

After combining paths with branching ( 1-17, 18-22, 23-24, 25-29 ) (Figure 9), we obtain equivalent time characteristics of the equivalent graph vertices (Table 5). Figure 9: Result of consecutive branch merging    Table 5  Time characteristics after combining 

Obtained time indices algorithm activity of the radio monitoring system operator gives the opportunity to find the probabilities of timely decision making by the operator at any point in time by expression ( 9 ). Taking into account that in solving the problem the operator has normal law distribution, the average deviation in time of taking the decision by the operator will be found by expression   m  Tmin  3

The obtained value of the probability of a timely decision-making by the operator must be compared with the admissible value. The permissible value can be obtained in further research in relation to a specific task performed by the system.

So, the method for evaluation the probability of timely decision-making by the information system operator will consist of the following steps: 1. Make a probabilistic oriented graph of the functioning of the ergatiс information system. 2. Determine the timing of individual operations – Tmini , Tai , mi , Tmaxi .

3. Perform transformations of the graph using the rules of combining paths without branches and with branches and calculate the time indices – Tmin ,  Ta ,  m ,  Tmax .

4. Calculate the average deviation of the operator's decision time by expression (12). 5. Calculate the probability of timely decision-making by the operator of the information system by expression (9).

3.2. Method for evaluation the probability of error-free decision-making

To calculate the probability of error-free decision making by an information system operator we will use a method based on information analysis of decision making processes [21, 22]. The basis of the informational approach is the fact that the most essential factor, which influences the error-free decision making, is the volume of initial information. Accordingly, for any complex system, an increase in the amount of initial information leads to an increase in the probability of error-free decision making by an operator. Taking into account loss of initial information due to decrease of operator's reliability, let us write the expression for probability of error-free decision – Pd of the i -th operation:

Pdi  1  Y0e , where Iin – amount of initial information; Il – amount of information lost; Y0 – initial uncertainty of the decision taken characterizes the psychological confidence of the operator (at the optimal state of the operator Y0  1 ); Ki – coefficient that characterizes the value an information and affects the growth rate of probability depending on the amount of processed information by the operator. At maximum value of the information Ki tends to 0, at minimum – Ki = 1. The value of this coefficient will be found on the assumption that as the number of possible states of the system, and consequently the number of variants of decisions n , grows, the value of the information by the expression 1 Ki 

n

The amount of initial information about some system is equal to the entropy of this system

Iin  N , and at equal-probability n states of the system [17]

Iin  log n. (15) 

Over time, due to the impact of various reasons, part the original information will be lost, the value of the loss will depend on the level efficiency of the operator and determined by the introduced coefficient decrease in the reliability of the operator – K R . The amount is lost information we will find by the expression.

Il  Iin .K R . (16) 

The level of reliability will also affect the level in psychological confidence, that is, the coefficient

Y0 will decrease in accordance with the value of K R .

With the decrease of the level a reliability operators value of information will decrease. To take this into account let's write expression (14) in the form: ( 13 )  (14)  (17) 

Given expressions ( 13 ), (15), (16), (17), and assuming that the operator solution is completely indeterminate is Y0 =1, we obtain an expression for determining the probability of error-free solution at the i -th operation at an arbitrary time moment:  ni ( 1 K R )log ni

K R

Pdi  1  K Re .

To carry out calculations on the example the radio monitoring system, by expert polling was established the number from the i -th operation, determining the number of states – n (Table 6,

Table 7). In the table the numbers of operations correspond to the numbers from Table 1.

  Table 6  Time characteristics after combining  № operation 

Number of  possible solutions  1  3    Table 7  Time characteristics after combining 

№ operation  Number of possible  solutions  16  9 

To obtain the probability for error-free decision by the operator of the radio-monitoring system, was used probabilistic directed graph, represented on Figure 2. In the process of transforming the initial graph-scheme to the equivalent one, with one vertex, the union of the same paths was carried out, as in determining the time characteristics, taking into account the rules for transforming the graph [20]. It was taken into account that when merging paths without branching (Figure 10) probability characteristics are determined by the expression [20]

When combining tracks with branching (Figure 11)

1 Pd1 2 Pd2

….

n

Pd  i1 Pdi .

P1 1 p 1-p 2 3

P2 P3 n Pdn P4 4 probability characteristics are determined by the expression (18)  (19)  Pd  P1P4 ( pP2  ( 1  p )P3 ). (20) 

So, the method for evaluation the probability of error-free decision-making by the information system operator will consist of the following steps: 1. Make a probabilistic oriented graph for functioning of an ergatiс information system. 2. Determine the number or possible solutions of the i -th operation determining the number of states – n .

3. Calculate the probability for error-free decision by the operator when performing the i -th operation at an arbitrary moment of time by expression (18).

4. To carry out transformations to the graph using the rules of combining paths without branching and with branching and to carry out calculation – Pd by expressions (19), (20).

In order to verify the proposed methods for evaluation the performance of the ergatic information system, the probabilities involved in the timely decision-making and the probability of error-free decision-making by the operator information system for the range of changes in the coefficient K R  0  1 were calculated. The results are shown in Table 8, for which the corresponding dependencies were built (Figure 12).   Table 8  Calculation results 

K R  

Analysis of the data obtained shows that at the initial stage of reducing the operator's reliability (up to K R = 0.4), the performance indicators of the system functioning practically do not change. This can be explained by the inclusion of human compensatory mechanisms. After a certain value, a sharp decrease in the performance indicators of the system functioning to critical values is observed.

4. Conclusions 

The developed methods constitute the content of the generalized methodology for evaluation the performance indicators for ergatic information system, based on an algorithmic model of the operator's activity. The methods take into account the reduction factor of operator reliability, made it possible to obtain an estimate of probabilities for timely decision-making and probability of error-free decision-making by the information system operator in real time while on duty.

5. References 

Performance. Wright-Patterson AFB (OH): U.S. Air Force School of Aerospace Medicine, 2015.

Technical Report AFRL-SA-WP-SR-2015-0025. 14 p.

[14] A. Noroozian; A. Sayyah, A new hybrid reliability allocation method, based on developed MEOWA method, Birnbaum importance measure and fuzzy DEMATEL technique, International Journal of Critical Infrastructures (IJCIS), Vol. 16, No. 2. (2020) 162–185. doi:10.1504/IJCIS.2020.107269. [15] D. Kwon, A. Lucero, Optimized reliability test design to reduce uncertainties in reliability assessment in: 62nd. Electronic Components and Technology Conference, San Diego, CA, USA. (2012) 1726–1730, doi: 10.1109/ECTC.2012.6249070. [16] J. Lee, M. Mitici, An integrated assessment of safety and efficiency of aircraft maintenance strategies using agent-based modelling and stochastic Petri nets, Reliability Engineering & System Safety, Vol. 202, 107052 (2020) doi:10.1016/j.ress.2020.107052. [17] Werner L. Probability Theory. De Gruyter Textbook (2016) 395 p. doi:10.1515/9783110466195. [18] I. Korobiichuk, A. Tokar, Y. Danik, V. Katuha, in: Evaluation methods for the ergatic system reliability operator. Automation 2019: progress in automation, robotics and measurement techniques, Vol. 920 (2020) 560–570. doi:10.1007/978-3-030-13273-6_52. [19] L. Ying, G. Wen-xiang, A plan recognition algorithm based on the probabilistic goal graph, in: 2011 International Conference on Network Computing and Information Security, Guilin, 359– 362. doi:10.1109/ncis.2011.79. [20] Graph Theory, in: Probabilistic Networks and Expert Systems. Information Science and

Statistics. Springer, New York, NY (1999) doi:10.1007/0-387-22630-3_4. [21] M. Bar Eli, H. Plessner, M. Raab, Theories of Decision Making, in Judgement, Decision Making and Success in Sport (2011) 27–38. doi: 10.1002/9781119977032.ch3. [22] J. Mishra, D. Allen, A. Pearman, Information seeking, use, and decision making, Journal of the Association for Information Science and Technology, vol. 66( 4 ). (2015) 662–673. doi: 10.1002/asi.23204.

References

[1] T. Kletz , P. Amyotte , Accidents said to be due to human error, What Went Wrong? , 6th. ed. Butterworth-Heinemann. ( 2019 ) 53 - 172 . doi: 10 .1016/B978-0 -12-810539-9 . 00007 - 0 .

[2] B. Strauch , Investigating Human Error: Incidents, Accidents, and Complex Systems , 1st. ed., Routledge ( 2018 ) doi:10.4324/9781315251851.

[3] Y. Tan , D. Feng , H. Shen, Research for Unmanned Aerial Vehicle components reliability evaluation model considering the influences of human factors , 3rd International Conference on Mechanical, Electronic and Information Technology Engineering ., ICMITE 2017 , vol. 139 , article number 00221. doi: 10 .1051/matecconf/201713900221.

[4] Z. Mian , D. Mavris , Mission effectiveness quantification and assessment of Micro Autonomous Systems and Technologies , IEEE Systems and Information Engineering Design Symposium ( 2013 ) doi:10.1109/SIEDS. 2013 . 6549514 .

[5] Haifeng Zhu , Fundamental Models for Missions of Engineered Systems , Systems Conference (SysCon) IEEE International. ( 2019 ) 1 - 6 . doi: 10 .1109/syscon. 2019 . 8836714 .

[6] D. Liu , M. Jaramillo , D. Vincenzi , The effects of system reliability and task uncertainty on autonomous unmanned aerial vehicle operator performance under high time pressure . Hum. Factors Ergon . Vol. 25 , № 5 . ( 2015 ) 515 - 522 . doi: 10 .1002/hfm.20565.

[7] X. Pan , Z. Wu , Performance shaping factors in the human error probability modification of human reliability analysis , International Journal of Occupational Safety and Ergonomics , Vol. 26 . ( 2020 ) 538 - 550 , doi:10.1080/10803548. 2018 . 1498655 .

[8] Y. Rejeki , E. Achiraeniwati , A. Wanda , Measurement of operator reliability level using the Human Error Assessment and Reduction Technique (HEART) method . IOP Conference Series: Materials Science and Engineering , Vol. 830 , 2020 , pages 032095 , doi:10.1088/ 1757 - 899X/830/3/032095.

[9] D. Safitri , R. Astriaty , N. Rizani , Human Reliability Assessment dengan Metode Human Error Assessment and Reduction Technique pada Operator Stasiun Shroud PT .X, Jurnal Rekayasa Sistem Industri Vol. 4 , No. 1 , ( 2015 ) 1 - 7 . doi: 10 .26593/jrsi.v4i1. 1388 .1- 7 .

[10] H. Ahmad , M. Faizal , M. Mazleha, Research reactor operator performance based on the human error assessment and reduction technique (HEART) , in IOP Conference Series: Materials Science and Engineering , Vol. 785 , International Nuclear Science Technology and Engineering Conference 2019 ( iNuSTEC2019 ) 29 - 31 October 2019 , Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia. doi: 10 .1088/ 1757 -899x/785/1/012048.

[11] H. Che , S. Zeng , J. Guo , Reliability assessment of man-machine systems subject to mutually dependent machine degradation and human errors . Reliab Eng Syst , Vol. 190 . ( 2019 ) 106504 . doi: 10 .1016/j.ress. 2019 . 106504 .

[12] Y. Fan , Study of a new Analysis Method of Risk Priority Number Based on FMEA , SAE Technical Paper 2015 - 01 - 0136 . ( 2015 ) doi:10.4271/2015-01-0136.

[13] W. Chappelle , T. Goodman , J. Swearingen , W. Thompson . A Preliminary Investigation into Cognitive Aptitudes Predictive of Overall MQ-1 Predator Pilot Qualification Training