=Paper=
{{Paper
|id=Vol-2269/FSS-18_paper_3
|storemode=property
|title=Inefficiencies in Cyber-Security Exercises Life-Cycle: A Position Paper
|pdfUrl=https://ceur-ws.org/Vol-2269/FSS-18_paper_3.pdf
|volume=Vol-2269
|authors= Muhammad Mudassar Yamin,Basel Katt
|dblpUrl=https://dblp.org/rec/conf/aaaifs/YaminK18
}}
==Inefficiencies in Cyber-Security Exercises Life-Cycle: A Position Paper==
https://ceur-ws.org/Vol-2269/FSS-18_paper_3.pdf
Inefficiencies in Cyber-Security Exercises Life-Cycle: A Position Paper
Muhammad Mudassar Yamin and Basel Katt
Department of Information Security and Communication Technology
Norwegian University of Science and Technology (NTNU), Norway
muhammad.m.yamin@ntnu.no, basel.katt@ntnu.no
Abstract
Our world is becoming digitalized day by day, this
leads to an increase amount of cyber-attacks by cyber-
criminals. To tackle the increasing amount of cyber-
attacks, cyber-security professionals are required in a
high number. However, the required number of cyber-
security professionals is not present. Despite the fact
that academia and industry are trying to increase the
number of cyber-security professionals, however, the
tools and techniques used for cyber-security profes-
sional development are ineffective, as the gap between
required and available cyber-security professionals is
still increasing. One of the primary tools that is used
in cyber-security professional development is hands-on
cyber-security exercises. In this position paper, we will
analyze the inefficiencies present in conducting hands-
on cyber-security exercises and what can be done to Figure 1: Participants knowledge test prior and after
reduce and eliminate those inefficiencies. cyber-security exercise (Moore, Fulton, and Likarish
2017)
INTRODUCTION
Cyber-security exercises run attack and defense scenar-
ios on a virtual and physical environment. A team months (Vykopal et al. 2017). This makes cyber-
of individuals, known as white time, creates the en- security exercises very costly and time consuming to
vironment. In the environment, a team of attack- be used in large scale to help reducing the growing
ers, known as red team, tries to exploit vulnerabilities cyber-security skills gap (Furnell, Fischer, and Finch
present in the environment while a team of defenders, 2017). Researchers divided cyber-security exercise in
known as a blue team, tries to defend and prevent the five phases to get the clear picture of cyber-security
attacks. In a recent study (Moore, Fulton, and Likar- exercise development and execution steps. These five
ish 2017) researchers find out that such an exercise phases make the cyber-security exercise development
is very beneficial in cyber-security skill development. and execution life cycle (Vykopal et al. 2017):
The researcher conducted knowledge surveys on par- • Preparation It is the lengthiest part of cyber-
ticipants before and after a cyber-security exercise and security exercise development and execution. It in-
they found significant improvement in network security volves setting up exercise objectives, defining a story,
skills like ARP-Posioning, duplication in DNS entries establishing points weight-age and creating a virtual
and firewall/routers assessment as seen in figure 1. environment for the cyber security exercise.
These cyber-security exercises are usually conducted
within hours and days but the time required to pre- • Dry run The dry run is the testing of the devel-
pare these cyber-security exercises often spans up to oped virtual environment according to exercise ob-
jective by cyber-security experts. This process also
Copyright c by the papers authors. Copying permitted takes long time due to the changes and adjustments
for private and academic purposes. In: Joseph Collins, required for the cyber-security exercise.
Prithviraj Dasgupta, Ranjeev Mittu (eds.): Proceedings of
the AAAI Fall 2018 Symposium on Adversary-Aware Learn- • Execution This is the phase where the actual cyber-
ing Techniques and Trends in Cybersecurity, Arlington, VA, security exercise takes place. Teams of attackers and
USA, 18-19 October, 2018, published at http://ceur-ws.org defenders try to achieve the set of defined objectives.
Based upon the complexity of cyber-security exercise Michigan cyber-range (Jones et al. 2015). SVED on the
it can take hours to days. other hand utilizes freely available exploit tools such as
• Evaluation At this phase teams performance is as- metasploit and nmap and automate their operations to
sessed according to the level of exercise objective execute red team activities in a cyber security exercise.
completion. Feedback from participants is collected SVED is deployed at CRATE cyber range (Sommes-
for future exercises. This phase usually takes few tad 2015). In term of cyber-defense process execution
hours for its completion. it is identified that in a cyber-security exercise skilled
human professionals are required to conduct the ex-
• Repetition The whole process is repeated for a set ercise. Most of the cyber-defense research is focused
of new teams utilizing the lessons learned from the on antivirus, antimalware, firewall and SIEM develop-
previous exercises and making necessary changes. ment, which left a lot of room for improvement in cyber-
Conducting cyber security exercises in the described defense process execution without human involvement
manner is a length, tedious and error-prone pro- in a cyber-security exercise.
cess (Beuran et al. 2018). Therefore, it is the posi- In term of evaluation phasen in most cyber-security
tion put forward that cyber-security exercises are a good exercises the theme of the exercise is CTF (Capture
tool for cyber-security skill development but the ineffi- the flag competition). As the name suggests flags are
ciencies in cyber-security exercise development and ex- used for point scoring and evaluation purposes. Flags
ecution life cycle limits its ability to be widely used for contain some value of a random length when submitted
cyber-security skill development. to exercise or competition management systems points
will be awarded. Based upon the number of points
SURVEY OF THE LITERATURE at the end of cyber security exercise or CTF competi-
Researchers have been trying to reduce the inefficiencies tion, teams are evaluated. But the flag based evaluation
present in conducting cyber-security exercise. In term mechanism is not ideal for overall performance analysis
of environment preparation phase of cyber-security ex- of individuals and teams. Flags only indicate that they
ercise life cycle most of the current research is fo- either successful or not in completing a task, flags dont
cused on reducing the time required for the preparation indicate at which approach they use or at which stage
of virtual environment. Researchers developed multi- they feel difficult in completing the task. To tackle
ple solutions for this problem two of them are Tele- this problem KYPO (Čeleda et al. 2015) cyber range
lab (Willems and Meinel 2012) and SecGen (Schreuders implemented an evaluation mechanism that is depen-
et al. 2017) (security scenario generator). In TeleLab dent upon event log monitoring. Event logs contains
the researchers created multiple templates of virtual specific information about the activities that are being
environment and developed an environment definition performed on a system. Based upon this information
language, through which existing template are modi- automatic evaluation is performed.
fied automatically to create new virtual environments
for cyber security exercises. In SecGen researchers take ANALYSIS AND DISCUSSION
the approach of TeleLab a bit further. Instead of defin- The literature contains interesting solutions for the re-
ing the detailed environment schema using an environ- duction of inefficiencies in cyber-security exercise de-
ment definition language, SecGen takes the environ- velopment and execution life cycle. But these solutions
ment requirement i.e. number of machine, number of have their cons as well. If we consider the autonomous
vulnerabilities, type of vulnerabilities etc. as input and generation of experimental environment by TeleLab and
randomly generate a virtual environment through the SecGen, we will notice that the environment which is
combination of existing virtual environment templates. generated is based upon an environment that is already
In the dry-run phase of cyber-security exercise recent available and if a participant already participated in an
studies (Ošlejšek et al. 2018) has shown that this phase environment that is used for the creation of the environ-
has a lot of room for improvement. It is identified that ment then the participants will have unfair advantage.
team of human attacker and defenders does manual ver- The autonomous attack execution in the cyber-
ification of the developed environment. That makes the security exercise by SC2RAM and SVED gives a ca-
process quite inefficient. pability to the team of defenders to practice their skills
In execution phase multiple solutions in the litera- without the availability of an actual attacker. But these
ture are available for the execution of cyber-attacks tools are currently at an initial phase of their testing
and defense in cyber-security exercise execution. Two and have only basic capabilities. That makes them un-
of the attack execution tools are Simulated Cognitive suitable for realistic training.
Cyber Red-team Attack Agent (SC2RAM) (Jones et The scoring mechanism in KYPO cyber range is a
al. 2015) and Scanning, Vulnerabilities, Exploits and very good approach for automatic evaluation of a par-
Detection tool (SVED) (Holm and Sommestad 2016). ticipants performances in a cyber security-exercsie by
SC2RAM is developed to mimic the red team execution monitoring the event logs created by the participants
steps in a cyber security-exercise. It can perform ba- activity. However, this approach can only give a holis-
sic DoS (Denialof-Service) attack on a given network. tic view of participant performance, which is only good
It is still at prototyping stage and is being tested at for calculating the overall performance of a participant,
not the performance of a participant at specific phase life cycle .This will (1) reduce the cost and time re-
of the cyber-security exercise. quire for conducting cyber-security exercise, (2) provide
better training by always-available autonomous adver-
POTENTIAL SOLUTION saries, and (3) make cyber-exercises computationally
repeatable for conducting systematic training.
Research is being carried out to address the issues
present in conducting operation based cyber-security
exercises. Researchers in (Jones et al. 2015) presented
References
a novel technique to model and execute an active op- Beuran, R.; Tang, D.; Pham, C.; Chinen, K.-i.; Tan,
position in a cyber-security exercise. The researchers Y.; and Shinoda, Y. 2018. Integrated framework for
discussed the missing element in the exercise environ- hands-on cybersecurity training: Cytrone. Computers
ment that is active opposition. The researchers argued & Security.
that:The environment may have static defenses, such Čeleda, P.; Čegan, J.; Vykopal, J.; and Tovarňák,
as access control or firewalls, or a fixed set of intrusion D. 2015. Kypo–a platform for cyber defence exer-
methods to defend against, but it typically lacks any ac- cises. M&S Support to Operational Tasks Including War
tive opposition that might adapt defensive or offensive Gaming, Logistics, Cyber Defence. NATO Science and
actions (e.g., monitor logs, blocked connections, exploit Technology Organization.
switching or information gathering) Furnell, S.; Fischer, P.; and Finch, A. 2017. Can’t get
The researchers presented techniques to model cyber- the staff? the growing need for cyber-security skills.
attack/defense adversaries and highlighted possible ap- Computer Fraud & Security 2017(2):5–10.
proaches that can be used in the implementation of Holm, H., and Sommestad, T. 2016. Sved: Scan-
such adversaries. Based upon this research, a tool is ning, vulnerabilities, exploits and detection. In Mil-
developed for autonomous execution of highly skilled itary Communications Conference, MILCOM 2016-
red-team attackers SC2RAM: A Deployable Cognitive 2016 IEEE, 976–981. IEEE.
Model of a Cyber Attacker (Jones et al. 2015). This tool
can train blue teamers to tackle cyber-security chal- Jones, R. M.; OGrady, R.; Nicholson, D.; Hoffman,
lenges and can configure and test defensive systems. R.; Bunch, L.; Bradshaw, J.; and Bolton, A. 2015.
SC2RAM is deployed at Michigan cyber-range to per- Modeling and integrating cognitive agents within the
form basic cyber-attack simulation, as it is still at pro- emerging cyber domain. In Proceedings of the Inter-
totype stage. On the other hand tools that mimic blue service/Industry Training, Simulation, and Education
teams actions in a security exercise is still need to be Conference (I/ITSEC), volume 20. Citeseer.
implemented (Jones et al. 2015). We are planning to Moore, E.; Fulton, S.; and Likarish, D. 2017. Eval-
model the roles of white, blue and red teamers with uating a multi agency cyber security training program
respect to each other for the development of a cyber- using pre-post event assessment and longitudinal analy-
security exercise platform that can assist execution of sis. In IFIP World Conference on Information Security
cyber-security exercises in a autonomous manner by Education, 147–156. Springer.
autonomously preparing the exercise environment and Ošlejšek, R.; Vykopal, J.; Burská, K.; and Rusňák, V.
generating autonomous adversaries according to the ex- 2018. Evaluation of cyber defense exercises using visual
ercise environment. This will effectively remove the analytics process.
need of human adversaries and support staff required Schreuders, Z. C.; Shaw, T.; Ravichandran, G.; Keigh-
for conducting a cyber-security exercise. By reducing ley, J.; Ordean, M.; et al. 2017. Security scenario gen-
these inefficiencies cyber-security exercises can be con- erator (secgen): A framework for generating randomly
ducted regularly at a wider scale, which will help in vulnerable rich-scenario vms for learning computer se-
reducing the cyber-security skill gap currently present curity and hosting ctf events. In USENIX. USENIX
in industry. Association.
Sommestad, T. 2015. Experimentation on operational
CONCLUSIONS cyber security in crate. NATO STO-MP-IST-133 Spe-
From the above discussion it can be observed that mul- cialist Meeting, Copenhagen, Denmark.
tiple phases involved in cyber-security exercise devel- Vykopal, J.; Vizváry, M.; Oslejsek, R.; Celeda, P.; and
opment and execution can be automated to reduce cost Tovarnak, D. 2017. Lessons learned from complex
and time required for conducting cyber-security exer- hands-on defence exercises in a cyber range. In Fron-
cises in an efficient manner. As it was suggested ear- tiers in Education Conference (FIE), 1–8. IEEE.
lier inefficiencies in cyber-security exercise development
and execution life cycle limit its ability to be widely used Willems, C., and Meinel, C. 2012. Online assessment
for cyber-security skill development. We can conclude for hands-on cyber security training in a virtual lab. In
that the roles of white, blue and red teamer in a cyber- Global Engineering Education Conference (EDUCON),
security exercise need to be executed autonomously, 2012 IEEE, 1–10. IEEE.
which will increase the efficiency of preparation, exe-
cution and evaluation phases in cyber-security exercise