Intelligent Method for CSIRT Performance Evaluation in Critical Information Infrastructure

Abstract

In this paper authors have developed a method for Computer Security Incident Response Team (CSIRT) performance evaluation, which is implemented in the following stages: determining the performance of the CSIRT, defining the Key Performance Indicators (KPI), building a panel of indicators. The developed method can be used to monitor, manage, analyze and enhance the effectiveness of the CSIRT in critical information infrastructure as well as in common (general) information and communication systems. The experimental study of developed method realization for domestic cellular provider was also presented. Given results can be useful for information security audit of company, region or state. Method and the tools based on it will be useful to the leaders of the cyber incident response centers for monitoring, analyzing, assessing and managing the effectiveness of the CSIRT. The developed method can be applied to any company or government agency in order to increase both the level of information security and the efficiency of the work of the employee, department and organization as a whole.

Keywords

• CSIRT
• KPI
• Correlation Matrix
• Efficiency
• Critical Information Infrastructure

-

1. Introduction Now, the information security of persons, societies and states is one of the main components of national security in general because information and communication technologies are widely used in all areas.

The problem of information security is not only actual, but also global. Information security incidents become more complex and often [1-4]. Usually, the response to cyber incident directed at CSIRT (Computer Security Incident Response Team) which every year receive more and more assignments and challenges [5]. It becomes necessary to evaluate and analyze the work of CSIRT [6]. This index is most important to informational security of some organization or country. Periodic (monthly, quarterly, etc.) evaluation of CSIRT`s work authorize strong and weak departments, groups, some employees for improving their work in future and highlight some trends based on statistical data. It has special importance in critical information infrastructure for example communication, transportation etc. The analysis showed that CSIRT performance evaluation not given enough attention, and this could adversely affect the level of information security. After analyzing, the existing methods for evaluating staff or unit discovered that none of the methods is universal. Everyone has advantages and disadvantages. In addition, in order to achieve the maximum result in the evaluation it is possible to use several methods simultaneously. Moreover should take into account the specifics of the organization, staff or unit is estimated. The chosen methods should meet to the structure of the enterprise, the nature of the activities of staff, the objectives of evaluation, to be simple and understandable; include both qualitative and quantitative indicators [7]. Based on this, has developed a method that combines the advantages of known techniques to minimize gaps and takes into account the specifics of the CSIRT. The developed method consists of three steps: determining the performance of the CSIRT, determining the key performance indicators of the CSIRT, building a panel of indicators and visualizing the dependence of Key Performance Indicators (KPI) and Efficiency (E). 2. Theoretical background and experimental study of proposed method Stage 1 – Determining the Performance of the CSIRT When a CSIRT is functioning, the information about Cyber incidents is recorded to the database (DB). Among the basic indicators of the functioning of CSIRT [8, 9], which have quantitative values should be allocated the following (described in following Table 1).

Mark E LRI INAI DRI ECS PRI DIR CII  p  PI   PI q   PI1, PI 2 ...., PI p  q1  ( 1 ) where PI q  PI , (q  1, p) , p , is the number of performance indicators of the CSIRT. Experimental study will include different input data for Ukrainian cellular provider (as a part of critical information infrastructure of the state, described in papers [5, 14, 16]) for all stages (accumulated statistics for 1st and 2nd quarters of 2018 in accordance – Variant 1 and Variant 2).

Variant 1 (1Q, 2018) For example, using database with CSIRT metrics for domestic cellular provider during 1st quarter of 2018, let’s form Table 2. E, LRI , INAI , DRI , ESC, PRI , DIR,CII  where PI1  PI E  E, PI 2  PI LRI  LRI ,.....PI8  PI CII  CII are metrics of CSIRT activity (performance).

Output data of this stage consist of metrics of CSIRT performance described in mentioned Table 2.

Variant 2 (2Q, 2018) For example, using database with CSIRT metrics for domestic cellular provider during 2nd quarter of 2018, let’s form Table 3. Using ( 1 ) and data from Table 1 when p  8 , we will get:

 8  PI celprov _ ua3   PI q   PI1, PI 2 ,.....PI 8  

q1  PI E , PI LRI , PI INAI , PI DRI , PI ESC , PI PRI , PI DIR , PI СП   E, LRI , INAI , DRI , ESC, PRI , DIR,CII  where

PI1  PI E  E, PI 2  PI LRI  LRI ,.....PI 8  PI CII  CII metrics of CSIRT performance.

In a similar way to variant 1 output data of this stage consist of metrics of CSIRT performance described in mentioned Table 3.

Stage 2 – Determination of Key Performance Indicators for CSIRT To determine the Key Performance Indicators from the set of CSIRT performance indicators was used the multiple correlation-regression analysis process [10], which includes the following steps: Step 1. Selection of all possible factors, which affect on the indicator (or process) that being investigated. Each factor determines numerical characteristics if some factors can`t be quantitatively or qualitatively determined or statistics are not available to them, they will removed from further consideration.

Step 2. Choosing a regressive or multi-factor model, that is finding an analytical expression that describes the link between factors with the resultant (function selection): 

Y  f ( x1, x2, x3, ......, xd )  where Y is resultant variable function; x x

1, 2, x3, ......, xd are factors signs.

An important problem is the choice of an analytical form for a function that links factors with a resultant feature-function. This function has to show real connections between the studied parameters and factors. It is important to note that the empirical justification of the type of function using the graphic analysis of the connections for multi-tasking models is unsuitable. Given that, any function of many variables by logarithms or replacement of variables can be reduced to a linear form then in practice the multiple regression equations are given linearly: 

Y  (a0 x0  a1x1  a2 x2  ...ad xd ) ( 3 ) where a0 , a1, a2 ..ad are parameters of the equation must to be measured. If for every factor and for a productive feature known d values yh , x1h , x2h ,...xdh at h  1,2,...., m then using the standard procedure of the least squares method to evaluate the parameters a system of linear algebraic equations will be obtained.  m m m m  a0 m  a1  x1 j  a2  x2 j  .....ad  xdj   y j  j1 j1 j1 j1 a0 m x1 j  a1 m x 21 j  a2 m x1 j x2 j  .....ad m x1 j xdj  m x1 j y j  j1 j1 j1 j1 j1  a0 m xdj  a1 m xdj x1 j  a2 m xdj x2 j  .....ad m x 2 dj  m xdj y j  j1 j1 j1 j1 j1 obtained system

d  1 of equations with d  1 unknowns a0 , a1,....ad solved by methods of linear algebra. For many equations would be best to use the ( 2 ) ( 4 ) The can be – Relative error of the residues and its average value: u yh h  h 100%,   h1 m m  h – RMS error variance disturbances: – Determination factor: method of choice Gauss main element. Since the matrix of the system of linear equations is symmetric, it is always a solution, and the only one. If the number of equations is small, then can be successfully used the inverse matrix method to solve the problem.

Step 3. Activity checking of received model. To do this need to calculate: – Remnants of the model as the differences between the observed and estimated values:

 uh  yh  yh  yh  (a0  a1x1h  a2 x2h  ....  ad xdh ), h  1,2,..., m ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 )  u  m  uh2 h1 m  d 1 m  uh2 h1 m  ( yh  y) 2 h1 m  ( yh  y) 2 or R 2  1  h1 m  ( yh  y) 2 h1 R 2  1  R  m  ( yh  y)2 1  h1 m  ( yh  y)2 h1 – Coefficient of multiple correlation, which is the main indicator of the correlation density of a generalized indicator with factors: All values of the coefficient of correlation R belong to the interval from -1 to 1. The sign of the coefficient shows the «direction» of the connection: the positive value indicates a "direct" connection, the negative value – about the «reverse» connection, and the value «0» – the absence of linear correlation communication. With R  1 or R  1 system has functional link between the signs. The multiplicity of the correlation coefficient is the main characteristic of the tightness of the link between the resultant sign and the combination of factors.

F  h1 m   ( yh  y) 2

d h1 m ( yh  yh ) 2 m  d 1 or F 

R 2 1  R 2  m  d 1 d Step 4. Checking the statistical significance of the results. Testing is carried out using Fisher statistics with d and (m  d 1) degrees of freedom: ( 10 ) ( 11 )  where d is the number of factors included in the model; m is total number; yh is estimated value of the dependent variable at h-th observation; y is the average value of the dependent variable; observation; R is coefficient of multiple correlation.

According to Fisher's tables critical value Fкр at d yh is the value of the dependent variable at h-th and (m  d 1) degrees of freedom. If F  Fкр , it is means about adequacy of the constructed model. If the model is not adequate then it is necessary to return to the stage of constructing the model and possibly introduce additional factors or switch to a nonlinear model. Step 5. Check significance of regression coefficients. Testing is carried out using tstatistics that parameters for multivariate regression is: ah th   a2h where ah is standard deviation assessment of h parameter.

If the value of th exceeds the critical value, which is based on the tables of the tcriterion of the Student, then the corresponding parameter is statistically significant and has a significant impact on the aggregate indicator.

Step 6. Calculation the elasticity factor. Differences in the units of measurement of factors are eliminated by using partial elasticity factors, which are given by the ratio:  h  dy  xh ( 12 ) dxh y where xh is average value of h-th parameter; y is the average value of effective signs. Partial elasticity coefficient indicates the percentage change in average productive sign of a change of 1% factor for fixed values of other parameters.

Step 7. Determination of confidence intervals for regression parameters. Confidence interval at reliability level (1-  ) is an interval with randomly defined limits with confidence level (1-  ) Overstate the true value of the coefficient of the regression equation ah and has the following form: ah  ta / 2,z a2h ; ah  ta / 2,z a2h (13) where ta / 2,z is Student`s statistics with z  m  d 1 degrees of freedom and levels of significance  ; a2h is average square deviation of estimation parameter ah . Suppose system has s random variables x1, x2 ,....., xrz ,....., xrv (investigated parameters) represented by samples by v values xr  xr1, xr 2 ,, xrz ,, xrv  For each pair of random variables xr and xw the equation can estimate the value of the empirical coefficient of linear correlation rrw . The obtained coefficients are written into the matrix size S  S : All correlation coefficient r belong to the interval from -1 to 1. The sign of the coefficient shows the «direction» of the connection: the positive value indicates a «direct» connection, the negative value – about the «reverse» connection, and the value «0» – the absence of linear correlation communication. With R  1 or R  1 system has functional link between the signs. The multiplicity of the correlation coefficient is the main characteristic of the tightness of the link between the resultant sign and the combination of factors. [11].

Using the above calculation procedure of multiple regression analysis it is possible to evaluate the degree of influence on the researched result indicator PI1 each of the factors introduced into the model PI 2 , PI 3 ,....PI p and identify a set of KPI:  av  KPI    KPI aw   KPI1 , KPI 2 ,....., KPI av 

aw1  where KPI aw  KPI , (aw  1, av) is number of KPI.

Variant 1 (1Q, 2018) Input data for current stage consist of matrix with CSIRT performance metrics (Table 2). Next by using multiple correlation-regression analysis we will get correlation matrix (Table 4). Analyzing mentioned Table 4 and using Chaddock's scale we can declare about the most influence factors: The priority of the incident (PRI); Incorrect number appointments of the incident (INAI); Evaluation customer satisfaction (ECS); Information provided about the incident (CII).

Output data of this stage in accordance to ( 2 ) and when w  4 is the following set of Key Performance Indicators KPI :

 4  KPI CSIRT1Q   KPI w   KPI1, KPI 2, KPI 3, KPI 4 

w1  KPI PRI , KPI INAI , KPI ESC, KPI CII  PRI , INAI , ESC, CII , where

KPI1  KPI PRI  PRI , KPI 2  KPI INAI  INAI , KPI 3  KPI ESC  ESC, KPI 4  KPI CII  CII

are Key Performance Indicators: the priority of the incident, incorrect number appointments of the incident, evaluation customer satisfaction, information provided about the incident consequently.

Variant 2 (2Q, 2018) Input data for current stage consist of matrix with CSIRT performance metrics (Table 3). Next by using multiple correlation-regression analysis we will get correlation matrix (Table 5).

Analyzing mentioned Table 5 and using Chaddock's scale we can declare about the most influence factors: The priority of the incident (PRI); Incorrect number appointments of the incident (INAI); Information provided about the incident (CII). In similar manner to variant 1 output data of this stage in accordance to ( 2 ) and when w  3 is the following set of Key Performance Indicators KPI :

 3  KPI CSIRT 2Q   KPI w   KPI1, KPI 2, KPI 3, 

w1  KPI PRI , KPI INAI , KPI CII  PRI , INAI , CII , where

KPI1  KPI PRI  PRI , KPI 2  KPI INAI  INAI , KPI 3  KPI CII  CII

are Key Performance Indicators: the priority of the incident, incorrect number appointments of the incident, information provided about the incident consequently. Stage 3 – Indicators Panel and Visualization for KPI and E Dependencies Proposed method is constricting the indicators panel [15], which will help with monitoring and CSIRT performance management. The indicators panel is tool for visualization and information analysis about business processes and their effectiveness. The data displayed on the panel indicators usually looks in the KPI form. Panel indicator system may be part of a corporate information system or act as a standalone application [12,15]. Using the indicator panel will present the data in a convenient form – diagrams, charts and data charts. For each organization, depending on its operational, planning and strategic tasks, this panel is made individually [13]. Variant 1 (1Q, 2018) Using output data from 2nd stage we can visualize given results, presented in Fig. 1-2: a)

b) c) d) Fig. 1. Correlation coefficients values: a) the priority of the incident; b) evaluation customer satisfaction; c) incorrect number appointments of the incident; d) information provided about the incident consequently.

E

Evaluation customer satisfaction

b)

Information provided about the incident consequently

c) E

The priority of the incident

d) Fig. 2. Efficiency dependency on: a) incorrect number appointments of the incident; b) evaluation customer satisfaction; c) information provided about the incident consequently; d) the priority of the incident Analysis of given results presented on Figs. 1-2 gives a possibility to define dependency between E and all of defined KPI and also form limitations: if INAI > 1, then E < 100; if ECS < 7, then E < 100; if CII < 60, then E < 100; if PRI < 2, then E < 100. Variant 2 (2Q, 2018) Using output data from 2nd stage we can visualize given results, presented in Fig. 3-4: a) b) c) Fig 3. Correlation coefficients values: a) the priority of the incident; b) information provided about the incident consequently; c) incorrect number appointments of the incident.

Incorrect number appointments of the incident

a) E

Information provided about the incident consequently

b) E

The priority of the incident

c) Fig. 4. Efficiency dependency on: a) Incorrect number appointments of the incident; b) Information provided about the incident consequently; c) The priority of the incident Analysis of given results presented on Fig. 3-4 gives a possibility to define dependency between E and all of defined KPI and also form limitations: if INAI > 1, then E < 100; if CII < 70, then E < 100; if PRI < 2, then E < 100.

Conclusions As can be seen from the theoretical background and experimental study, proposed method for assessing the effectiveness of the CSIRT can be used for determining the performance of the CSIRT. It allows the allocation of Key Performance Indicators among of them, using a multi-factor correlation-regression analysis in construction of indicators panel and visualization of KPI and Efficiency dependencies gives an opportunity to audit the CSIRT activities (performance) and other centers of information and telecommunication systems maintenance (particularly in critical information infrastructure). This method and the tools based on it will be useful to the incident response centers managers for monitoring, analyzing, assessing and managing the effectiveness of the CSIRT. Since the method is universal and can be applied to any company or government agency, in order to increase both the level of information security and the efficiency of the employee, department and organization.

13. Z. Hu, Yu. Khokhlachova, V. Sydorenko, I. Opirskyy, Method for Optimization of Information Security Systems Behavior under Conditions of Influences, International Journal of Intelligent Systems and Applications (IJISA), vol.9, № 12, 2017, pp.46-58.

14. S. Gnatyuk, V. Sydorenko, M. Aleksander, Unified data model for defining state critical information infrastructure in civil aviation, Proceedings of the 2018 IEEE 9th International Conference on Dependable Systems, Services and Technologies (DESSERT), Kyiv, Ukraine, May 24-27, 2018, pp. 37-42.

15. R. Odarchenko, V. Gnatyuk, S. Gnatyuk, A. Abakumova, Security Key Indicators Assessment for Modern Cellular Networks Kyiv, Proceedings of the 2018 IEEE First International Conference on System Analysis & Intelligent Computing (SAIC), Ukraine, October 8-12, 2018, pp. 1-7.

16. Yu. Danik, R. Hryschuk, S. Gnatyuk, Synergistic effects of information and cybernetic interaction in civil aviation, Aviation, vol. 20, №3, 2016, рр. 137-144.

References

1. Z. Hu , S. Gnatyuk , O. Koval , V. Gnatyuk , S. Bondarovets , Anomaly Detection System in Secure Cloud Computing Environment , International Journal of Computer Network and Information Security , Vol. 9 , № 4, рр. 10 - 21 , 2017 .

2. M. Aleksander , L. Dubchak , V. Chyzh , A. Naglik et al, Implementation technology software-defined networking in Wireless Sensor Networks , Proceedings of 2015 IEEE 8th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications , Warsaw, Poland, September 24-26 , 2015 .

3. Ya . Wahba, E. El Salamouny , Gh. El Taweel, Estimating the Sample Size for Training Intrusion Detection Systems , International Journal of Computer Network and Information Security (IJCNIS) , vol. 9 , No. 12 , pp. 1 - 10 , 2017 .

4. Security of Critical Information Infrastructures, Tobias Dehling, Sebastian Lins , Ali Sunyaev, Information Technology for Peace and Security , pp. 319 - 339 .

5. S. Gnatyuk , Critical Aviation Information Systems Cybersecurity , Meeting Security Challenges Through Data Analytics and Decision Support, NATO Science for Peace and Security Series, D : Information and Communication Security , IOS Press Ebooks, Vol. 47 , №3, рр. 308 - 316 , 2016 .

6. A. Gizun , V. Gnatyuk , N. Balyk , P. Falat , Approaches to Improve the Activity of Computer Incident Response Teams , Proceedings of the 2015 IEEE 8th International Conference on «Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications» (IDAACS' 2015 ), Warsaw, Poland, September 24-26 , 2015 : vol. 1 , pp. 442 - 447 , 2015 .

7. A. Tikhomirov , N. Kinash , S. Gnatyuk , A. Trufanov , O. Berestneva et al, Network Society: Aggregate Topological Models, Communications in Computer and Information Science . Verlag: Springer International Publ, vol. 487 , рр. 415 - 421 , 2014 .

8. Jan Van Bon, IT Service Management, 240 p., 2003 .

9. V. Kinzeryavyy , V. Gnatyuk , Basic performance parameters for cyberincidents response teams , Ukrainian Scientific Journal of Information Security , № 20 , № 2 , p. 193 - 196 , 2014 . DOI: 10 .18372/ 2225 - 5036 . 20 .7307

10. A. Marmoza, Theory of statistic, Кyiv, pp. 333 - 397 . 2013 .

11. Yu . Danik, R. Hryschuk , S. Gnatyuk , Synergistic effects of information and cybernetic interaction in civil aviation , Aviation , vol. 20 , №3, рр. 137 - 144 , 2016 .

12. Wayne W. Eckerson , Performance Dashboards, Moscow, Alpyna Business Books, 396 p., 2007 .