=Paper=
{{Paper
|id=Vol-2393/paper_362
|storemode=property
|title=Intelligent Method for CSIRT Performance Evaluation in Critical Information Infrastructure
|pdfUrl=https://ceur-ws.org/Vol-2393/paper_362.pdf
|volume=Vol-2393
|authors=Viktor Gnatyuk,Serhii Smirnov,Marek Aleksander,Liudmila Kharlai,Madina Bauyrzhan,Anzhelika Kokareva
|dblpUrl=https://dblp.org/rec/conf/icteri/GnatyukSAKBK19
}}
==Intelligent Method for CSIRT Performance Evaluation in Critical Information Infrastructure==
https://ceur-ws.org/Vol-2393/paper_362.pdf
Intelligent Method for CSIRT Performance Evaluation in
Critical Information Infrastructure
Viktor Gnatyuk1, Serhii Smirnov2, Marek Aleksander3, Liudmila Kharlai4,
Madina Bauyrzhan5, Anzhelika Kokareva6
1,6 National Aviation University, Kyiv, Ukraine
2 Central Ukrainian National Technical University, Kropivnitskiy, Ukraine
3 State University of Applied Sciences in Nowy Sącz, Nowy Sącz, Poland
4 Kyiv College of Communication, Kyiv, Ukraine
5 Satbayev University, Almaty, Kazakhstan
viktorgnatyuk@ukr.net smirnov.ser.81@gmail.com
aleksandermarek4@gmail.com lharlay@i.ua madina890218@gmail.com
lika_nk@ukr.net
Abstract. In this paper authors have developed a method for Computer Security
Incident Response Team (CSIRT) performance evaluation, which is
implemented in the following stages: determining the performance of the
CSIRT, defining the Key Performance Indicators (KPI), building a panel of
indicators. The developed method can be used to monitor, manage, analyze and
enhance the effectiveness of the CSIRT in critical information infrastructure as
well as in common (general) information and communication systems. The
experimental study of developed method realization for domestic cellular
provider was also presented. Given results can be useful for information
security audit of company, region or state. Method and the tools based on it will
be useful to the leaders of the cyber incident response centers for monitoring,
analyzing, assessing and managing the effectiveness of the CSIRT. The
developed method can be applied to any company or government agency in
order to increase both the level of information security and the efficiency of the
work of the employee, department and organization as a whole.
Keywords: CSIRT, KPI, Correlation Matrix, Efficiency, Critical Information
Infrastructure.
1. Introduction
Now, the information security of persons, societies and states is one of the main
components of national security in general because information and communication
technologies are widely used in all areas.
The problem of information security is not only actual, but also global. Information
security incidents become more complex and often [1-4]. Usually, the response to
cyber incident directed at CSIRT (Computer Security Incident Response Team) which
every year receive more and more assignments and challenges [5]. It becomes
necessary to evaluate and analyze the work of CSIRT [6]. This index is most
important to informational security of some organization or country. Periodic
(monthly, quarterly, etc.) evaluation of CSIRT`s work authorize strong and weak
departments, groups, some employees for improving their work in future and
highlight some trends based on statistical data. It has special importance in critical
information infrastructure for example communication, transportation etc.
The analysis showed that CSIRT performance evaluation not given enough attention,
and this could adversely affect the level of information security. After analyzing, the
existing methods for evaluating staff or unit discovered that none of the methods is
universal. Everyone has advantages and disadvantages. In addition, in order to
achieve the maximum result in the evaluation it is possible to use several methods
simultaneously. Moreover should take into account the specifics of the organization,
staff or unit is estimated. The chosen methods should meet to the structure of the
enterprise, the nature of the activities of staff, the objectives of evaluation, to be
simple and understandable; include both qualitative and quantitative indicators [7].
Based on this, has developed a method that combines the advantages of known
techniques to minimize gaps and takes into account the specifics of the CSIRT.
The developed method consists of three steps: determining the performance of the
CSIRT, determining the key performance indicators of the CSIRT, building a panel of
indicators and visualizing the dependence of Key Performance Indicators (KPI) and
Efficiency (E).
2. Theoretical background and experimental study of proposed
method
Stage 1 – Determining the Performance of the CSIRT
When a CSIRT is functioning, the information about Cyber incidents is recorded to
the database (DB). Among the basic indicators of the functioning of CSIRT [8, 9],
which have quantitative values should be allocated the following (described in
following Table 1).
Table 1. CSIRT performance indicators
Mark Name
E Efficiency
LRI Level of resolving the incident
INAI Incorrect number appointments of the incident
DRI Duration of resolving the incident
ECS Evaluation customer satisfaction
PRI The priority of the incident
DIR Duration of the incident registration
CII Information provided about the incident
For the implementation of this stage, use a set of performance indicators CSIRT PI
p
PI PI q PI 1 , PI 2 ...., PI p (1)
q 1
where PI q PI , ( q 1, p ) , p , is the number of performance indicators of the
CSIRT. Experimental study will include different input data for Ukrainian cellular
provider (as a part of critical information infrastructure of the state, described in
papers [5, 14, 16]) for all stages (accumulated statistics for 1st and 2nd quarters of
2018 in accordance – Variant 1 and Variant 2).
Variant 1 (1Q, 2018)
For example, using database with CSIRT metrics for domestic cellular provider
during 1st quarter of 2018, let’s form Table 2.
Table 2. Metrics of CSIRT performance during 1st quarters of 2018
№ E LRI INAI DRI ECS PRI DIR CII
1 90 4 3 1539 4 3 2 40
2 115 1 0 2502 8 4 6 80
… … … … … … … … …
600 171 1 0 37 6 1 6 85
Using (1) and data from Table 1 when p 8 , we will get:
8
PI celprov _ ua 2 PI q PI 1 , PI 2 ,.....PI 8
q 1
PI E , PI LRI , PI INAI , PI DRI , PI ESC , PI PRI , PI DIR , PI СП
E , LRI , INAI , DRI , ESC, PRI , DIR, CII
where PI1 PI E E , PI 2 PI LRI LRI ,.....PI 8 PI CII CII are metrics
of CSIRT activity (performance).
Output data of this stage consist of metrics of CSIRT performance described in
mentioned Table 2.
Variant 2 (2Q, 2018)
For example, using database with CSIRT metrics for domestic cellular provider
during 2nd quarter of 2018, let’s form Table 3.
Table 3. Metrics of CSIRT performance during 2nd quarters of 2018
№ E LRI INAI DRI ECS PRI DIR CII
1 109 1 0 55 5 1 5 70
2 86 4 2 560 4 2 4 60
… … … … … … … … …
600 150 1 0 40 8 1 4 60
Using (1) and data from Table 1 when p 8 , we will get:
8
PI celprov _ ua 3 PI q PI 1 , PI 2 ,.....PI 8
q 1
PI E , PI LRI , PI INAI , PI DRI , PI ESC , PI PRI , PI DIR , PI СП
E , LRI , INAI , DRI , ESC, PRI , DIR, CII
where PI1 PI E E , PI 2 PI LRI LRI ,.....PI 8 PI CII CII metrics of
CSIRT performance.
In a similar way to variant 1 output data of this stage consist of metrics of CSIRT
performance described in mentioned Table 3.
Stage 2 – Determination of Key Performance Indicators for CSIRT
To determine the Key Performance Indicators from the set of CSIRT performance
indicators was used the multiple correlation-regression analysis process [10], which
includes the following steps:
Step 1. Selection of all possible factors, which affect on the indicator (or process) that
being investigated. Each factor determines numerical characteristics if some factors
can`t be quantitatively or qualitatively determined or statistics are not available to
them, they will removed from further consideration.
Step 2. Choosing a regressive or multi-factor model, that is finding an analytical
expression that describes the link between factors with the resultant (function
selection):
Y f ( x1, x2, x3, ......, xd ) (2)
where Y is resultant variable function; x1, x2, x3, ......, xd are factors signs.
An important problem is the choice of an analytical form for a function that links
factors with a resultant feature-function. This function has to show real connections
between the studied parameters and factors. It is important to note that the empirical
justification of the type of function using the graphic analysis of the connections for
multi-tasking models is unsuitable. Given that, any function of many variables by
logarithms or replacement of variables can be reduced to a linear form then in practice
the multiple regression equations are given linearly:
Y (a0 x0 a1 x1 a2 x2 ...ad xd ) (3)
where a0 , a1 , a2 ..ad are parameters of the equation must to be measured.
If for every factor and for a productive feature known d values y h , x1h , x2 h ,...xdh at
h 1,2,...., m then using the standard procedure of the least squares method to
evaluate the parameters a system of linear algebraic equations will be obtained.
m m m m
a 0 m a 1 1j x a 2 2j x .....a d dj x yj
j 1 j 1 j 1 j 1
m m m m m
0 1j 1 2 1j 2 j d 1 j dj
2
a x a x 1 j a x x .....a x x x1 j y j
j 1 j 1 j 1 j 1 j 1 (4)
m m m m m
0 dj 1 dj 1 j 2 dj 2 j d
2
a x a x x a x x .....a x dj xdj y j
j 1 j 1 j 1 j 1 j 1
The
obtained system d 1 of equations with d 1 unknowns a0 , a1 ,....ad can be
solved by methods of linear algebra. For many equations would be best to use the
method of choice Gauss main element. Since the matrix of the system of linear
equations is symmetric, it is always a solution, and the only one. If the number of
equations is small, then can be successfully used the inverse matrix method to solve
the problem.
Step 3. Activity checking of received model. To do this need to calculate:
– Remnants of the model as the differences between the observed and estimated
values:
u h yh yh yh (a0 a1 x1h a2 x2 h .... ad xdh ), h 1,2,..., m (5)
– Relative error of the residues and its average value:
m
uh h
h 100%, h 1
(6)
yh m
– RMS error variance disturbances:
m
u 2
h
u h 1
(7)
m d 1
– Determination factor:
m m
2
uh2 ( y y)
h
R 2 1 m h1 or R 2 1 hm1 (8)
( yh y ) 2
h 1
( yh y ) 2
h 1
– Coefficient of multiple correlation, which is the main indicator of the correlation
density of a generalized indicator with factors:
m
2
( y y) h
R 1 hm1 (9)
( y y)
h 1
h
2
All values of the coefficient of correlation R belong to the interval from -1 to 1. The
sign of the coefficient shows the «direction» of the connection: the positive value
indicates a "direct" connection, the negative value – about the «reverse» connection,
and the value «0» – the absence of linear correlation communication. With R 1 or
R 1 system has functional link between the signs. The multiplicity of the
correlation coefficient is the main characteristic of the tightness of the link between
the resultant sign and the combination of factors.
Step 4. Checking the statistical significance of the results. Testing is carried out using
Fisher statistics with d and (m d 1) degrees of freedom:
m
( y y)
h 1
h
2
d R2 m d 1
F m or F (10)
1 R 2
( yh yh ) 2
d
h 1
m d 1
where d is the number of factors included in the model; m is total number; y h is
estimated value of the dependent variable at h-th observation; y is the average value
of the dependent variable; yh is the value of the dependent variable at h-th
observation; R is coefficient of multiple correlation.
According to Fisher's tables critical value Fкр at d and (m d 1) degrees of
freedom. If F Fкр , it is means about adequacy of the constructed model. If the
model is not adequate then it is necessary to return to the stage of constructing the
model and possibly introduce additional factors or switch to a nonlinear model.
Step 5. Check significance of regression coefficients. Testing is carried out using t-
statistics that parameters for multivariate regression is:
ah
th (11)
ah2
where ah is standard deviation assessment of h parameter.
If the value of t h exceeds the critical value, which is based on the tables of the t-
criterion of the Student, then the corresponding parameter is statistically significant
and has a significant impact on the aggregate indicator.
Step 6. Calculation the elasticity factor. Differences in the units of measurement of
factors are eliminated by using partial elasticity factors, which are given by the ratio:
dy xh
h (12)
dxh y
where x h is average value of h-th parameter; y is the average value of effective signs.
Partial elasticity coefficient indicates the percentage change in average productive
sign of a change of 1% factor for fixed values of other parameters.
Step 7. Determination of confidence intervals for regression parameters. Confidence
interval at reliability level (1- ) is an interval with randomly defined limits with
confidence level (1- ) Overstate the true value of the coefficient of the regression
equation a h and has the following form:
ah t a / 2, z ah
2
; ah t a / 2, z ah
2
(13)
where t a / 2 , z is Student`s statistics with z m d 1 degrees of freedom and levels
of significance ; ah is average square deviation of estimation parameter a h .
2
Suppose system has s random variables x1, x2 ,....., xrz ,....., xrv (investigated
parameters) represented by samples by v values xr xr1 , xr 2 ,, xrz ,, xrv For
each pair of random variables xr and x w the equation can estimate the value of the
empirical coefficient of linear correlation rrw . The obtained coefficients are written
into the matrix size S S :
1 k12 ... r1w ... k1s
r21 1 ... r2 w ... r2 s
. (14)
rr1 rr 2 ... 1 ... rrs
r rs 2 ... rsw ... 1
s1
All correlation coefficient r belong to the interval from -1 to 1. The sign of the
coefficient shows the «direction» of the connection: the positive value indicates a
«direct» connection, the negative value – about the «reverse» connection, and the
value «0» – the absence of linear correlation communication. With R 1 or R 1
system has functional link between the signs. The multiplicity of the correlation
coefficient is the main characteristic of the tightness of the link between the resultant
sign and the combination of factors. [11].
Using the above calculation procedure of multiple regression analysis it is possible to
evaluate the degree of influence on the researched result indicator PI 1 each of the
factors introduced into the model PI 2 , PI 3 ,....PI p and identify a set of KPI:
av
KPI KPI aw KPI 1 , KPI 2 ,....., KPI av (15)
aw1
where KPI aw KPI , (aw 1, av) is number of KPI.
Variant 1 (1Q, 2018)
Input data for current stage consist of matrix with CSIRT performance metrics
(Table 2). Next by using multiple correlation-regression analysis we will get
correlation matrix (Table 4).
Table 4. Correlation matrix for 1st quarters of 2018
Е 1
LRI -0,37 1
INAI -0,79 0,35 1
DRI -0,49 0,36 0,66 1
ECS 0,82 -0,47 -0,57 -0,51 1
PRI -0,91 0,57 0,47 0,67 -0,63 1
DIR 0,19 -0,52 -0,52 -0,63 0,43 -0,60 1
CII 0,89 -0,62 -0,52 -0,45 0,72 -0,58 0,29 1
Е LRI INAI DRI ECS PRI DIR CII
Analyzing mentioned Table 4 and using Chaddock's scale we can declare about the
most influence factors: The priority of the incident (PRI); Incorrect number
appointments of the incident (INAI); Evaluation customer satisfaction (ECS);
Information provided about the incident (CII).
Output data of this stage in accordance to (2) and when w 4 is the following set of
Key Performance Indicators KPI :
4
KPI CSIRT1Q KPI w KPI 1, KPI 2, KPI 3, KPI 4
w1
KPI PRI , KPI INAI , KPI ESC , KPI CII PRI , INAI , ESC, CII ,
where
KPI 1 KPI PRI PRI , KPI 2 KPI INAI INAI , KPI 3 KPI ESC ESC ,
KPI 4 KPI CII CII
are Key Performance Indicators: the priority of the incident, incorrect number
appointments of the incident, evaluation customer satisfaction, information provided
about the incident consequently.
Variant 2 (2Q, 2018)
Input data for current stage consist of matrix with CSIRT performance metrics (Table
3). Next by using multiple correlation-regression analysis we will get correlation
matrix (Table 5).
Analyzing mentioned Table 5 and using Chaddock's scale we can declare about the
most influence factors: The priority of the incident (PRI); Incorrect number
appointments of the incident (INAI); Information provided about the incident (CII).
Table 5.Correlation matrix for 2nd quarters of 2018
Е 1
LRI -0,35 1
INAI -0,83 0,43 1
DRI -0,43 0,38 0,62 1
ECS 0,64 -0,57 -0,52 -0,41 1
PRI -0,89 0,53 0,43 0,57 -0,53 1
DIR 0,23 -0,47 -0,56 -0,53 0,53 -0,45 1
CII 0,81 -0,54 -0,55 -0,55 0,62 -0,39 0,34 1
Е LRI INAI DRI ECS PRI DIR CII
In similar manner to variant 1 output data of this stage in accordance to (2) and when
w 3 is the following set of Key Performance Indicators KPI :
3
KPI CSIRT 2Q KPI w KPI 1, KPI 2, KPI 3,
w1
KPI PRI , KPI INAI , KPI CII PRI , INAI , CII ,
where KPI 1 KPI PRI PRI , KPI 2 KPI INAI INAI , KPI 3 KPI CII CII
are Key Performance Indicators: the priority of the incident, incorrect number
appointments of the incident, information provided about the incident consequently.
Stage 3 – Indicators Panel and Visualization for KPI and E Dependencies
Proposed method is constricting the indicators panel [15], which will help with
monitoring and CSIRT performance management. The indicators panel is tool for
visualization and information analysis about business processes and their
effectiveness. The data displayed on the panel indicators usually looks in the KPI
form. Panel indicator system may be part of a corporate information system or act as a
standalone application [12,15]. Using the indicator panel will present the data in a
convenient form – diagrams, charts and data charts. For each organization, depending
on its operational, planning and strategic tasks, this panel is made individually [13].
Variant 1 (1Q, 2018)
Using output data from 2nd stage we can visualize given results, presented in Fig. 1-2:
a) b)
c) d)
Fig. 1. Correlation coefficients values: a) the priority of the incident; b) evaluation
customer satisfaction; c) incorrect number appointments of the incident;
d) information provided about the incident consequently.
E
Incorrect number appointments of the incident
a)
E
Evaluation customer satisfaction
b)
E
Information provided about the incident consequently
c)
E
The priority of the incident
d)
Fig. 2. Efficiency dependency on: a) incorrect number appointments of the incident;
b) evaluation customer satisfaction; c) information provided about the incident
consequently; d) the priority of the incident
Analysis of given results presented on Figs. 1-2 gives a possibility to define dependency
between E and all of defined KPI and also form limitations: if INAI > 1, then E < 100; if
ECS < 7, then E < 100; if CII < 60, then E < 100; if PRI < 2, then E < 100.
Variant 2 (2Q, 2018)
Using output data from 2nd stage we can visualize given results, presented in Fig. 3-4:
a) b) c)
Fig 3. Correlation coefficients values: a) the priority of the incident; b) information provided
about the incident consequently; c) incorrect number appointments of the incident.
E
Incorrect number appointments of the incident
a)
E
Information provided about the incident consequently
b)
E
The priority of the incident
c)
Fig. 4. Efficiency dependency on: a) Incorrect number appointments of the incident;
b) Information provided about the incident consequently; c) The priority of the
incident
Analysis of given results presented on Fig. 3-4 gives a possibility to define
dependency between E and all of defined KPI and also form limitations: if INAI > 1,
then E < 100; if CII < 70, then E < 100; if PRI < 2, then E < 100.
3. Conclusions
As can be seen from the theoretical background and experimental study, proposed
method for assessing the effectiveness of the CSIRT can be used for determining the
performance of the CSIRT. It allows the allocation of Key Performance Indicators
among of them, using a multi-factor correlation-regression analysis in construction of
indicators panel and visualization of KPI and Efficiency dependencies gives an
opportunity to audit the CSIRT activities (performance) and other centers of
information and telecommunication systems maintenance (particularly in critical
information infrastructure). This method and the tools based on it will be useful to the
incident response centers managers for monitoring, analyzing, assessing and
managing the effectiveness of the CSIRT. Since the method is universal and can be
applied to any company or government agency, in order to increase both the level of
information security and the efficiency of the employee, department and organization.
References
1. Z. Hu, S. Gnatyuk, O. Koval, V. Gnatyuk, S. Bondarovets, Anomaly Detection System
in Secure Cloud Computing Environment, International Journal of Computer Network and
Information Security, Vol. 9, № 4, рр. 10-21, 2017.
2. M. Aleksander, L. Dubchak, V. Chyzh, A. Naglik et al, Implementation technology
software-defined networking in Wireless Sensor Networks, Proceedings of 2015 IEEE 8th
International Conference on Intelligent Data Acquisition and Advanced Computing Systems:
Technology and Applications, Warsaw, Poland, September 24-26, 2015.
3. Ya. Wahba, E. El Salamouny, Gh. El Taweel, Estimating the Sample Size for Training
Intrusion Detection Systems, International Journal of Computer Network and Information
Security (IJCNIS), vol.9, No.12, pp.1-10, 2017.
4. Security of Critical Information Infrastructures, Tobias Dehling, Sebastian Lins, Ali
Sunyaev, Information Technology for Peace and Security, pp. 319-339.
5. S. Gnatyuk, Critical Aviation Information Systems Cybersecurity, Meeting Security
Challenges Through Data Analytics and Decision Support, NATO Science for Peace and
Security Series, D: Information and Communication Security, IOS Press Ebooks, Vol.47, №3,
рр. 308-316, 2016.
6. A. Gizun, V. Gnatyuk, N. Balyk, P. Falat, Approaches to Improve the Activity of
Computer Incident Response Teams, Proceedings of the 2015 IEEE 8th International
Conference on «Intelligent Data Acquisition and Advanced Computing Systems: Technology
and Applications» (IDAACS’2015), Warsaw, Poland, September 24-26, 2015: vol. 1, pp. 442-
447, 2015.
7. A. Tikhomirov, N. Kinash, S. Gnatyuk, A. Trufanov, O. Berestneva et al, Network
Society: Aggregate Topological Models, Communications in Computer and Information
Science. Verlag: Springer International Publ, vol. 487, рр. 415-421, 2014.
8. Jan Van Bon, IT Service Management, 240 p., 2003.
9. V. Kinzeryavyy, V. Gnatyuk, Basic performance parameters for cyberincidents
response teams, Ukrainian Scientific Journal of Information Security, № 20, №2, p. 193-196,
2014. DOI: 10.18372/2225-5036.20.7307
10. A. Marmoza, Theory of statistic, Кyiv, pp. 333-397. 2013.
11. Yu. Danik, R. Hryschuk, S. Gnatyuk, Synergistic effects of information and cybernetic
interaction in civil aviation, Aviation, vol. 20, №3, рр. 137-144, 2016.
12. Wayne W. Eckerson, Performance Dashboards, Moscow, Alpyna Business Books,
396 p., 2007.
13. Z. Hu, Yu. Khokhlachova, V. Sydorenko, I. Opirskyy, Method for Optimization of
Information Security Systems Behavior under Conditions of Influences, International Journal of
Intelligent Systems and Applications (IJISA), vol.9, № 12, 2017, pp.46-58.
14. S. Gnatyuk, V. Sydorenko, M. Aleksander, Unified data model for defining state
critical information infrastructure in civil aviation, Proceedings of the 2018 IEEE 9th
International Conference on Dependable Systems, Services and Technologies (DESSERT),
Kyiv, Ukraine, May 24-27, 2018, pp. 37-42.
15. R. Odarchenko, V. Gnatyuk, S. Gnatyuk, A. Abakumova, Security Key Indicators
Assessment for Modern Cellular Networks Kyiv, Proceedings of the 2018 IEEE First
International Conference on System Analysis & Intelligent Computing (SAIC), Ukraine,
October 8-12, 2018, pp. 1-7.
16. Yu. Danik, R. Hryschuk, S. Gnatyuk, Synergistic effects of information and cybernetic
interaction in civil aviation, Aviation, vol. 20, №3, 2016, рр. 137-144.