Inefficiencies in Cyber-Security Exercises Life-Cycle: A Position Paper Muhammad Mudassar Yamin and Basel Katt Department of Information Security and Communication Technology Norwegian University of Science and Technology (NTNU), Norway muhammad.m.yamin@ntnu.no, basel.katt@ntnu.no Abstract Our world is becoming digitalized day by day, this leads to an increase amount of cyber-attacks by cyber- criminals. To tackle the increasing amount of cyber- attacks, cyber-security professionals are required in a high number. However, the required number of cyber- security professionals is not present. Despite the fact that academia and industry are trying to increase the number of cyber-security professionals, however, the tools and techniques used for cyber-security profes- sional development are ineffective, as the gap between required and available cyber-security professionals is still increasing. One of the primary tools that is used in cyber-security professional development is hands-on cyber-security exercises. In this position paper, we will analyze the inefficiencies present in conducting hands- on cyber-security exercises and what can be done to Figure 1: Participants knowledge test prior and after reduce and eliminate those inefficiencies. cyber-security exercise (Moore, Fulton, and Likarish 2017) INTRODUCTION Cyber-security exercises run attack and defense scenar- ios on a virtual and physical environment. A team months (Vykopal et al. 2017). This makes cyber- of individuals, known as white time, creates the en- security exercises very costly and time consuming to vironment. In the environment, a team of attack- be used in large scale to help reducing the growing ers, known as red team, tries to exploit vulnerabilities cyber-security skills gap (Furnell, Fischer, and Finch present in the environment while a team of defenders, 2017). Researchers divided cyber-security exercise in known as a blue team, tries to defend and prevent the five phases to get the clear picture of cyber-security attacks. In a recent study (Moore, Fulton, and Likar- exercise development and execution steps. These five ish 2017) researchers find out that such an exercise phases make the cyber-security exercise development is very beneficial in cyber-security skill development. and execution life cycle (Vykopal et al. 2017): The researcher conducted knowledge surveys on par- • Preparation It is the lengthiest part of cyber- ticipants before and after a cyber-security exercise and security exercise development and execution. It in- they found significant improvement in network security volves setting up exercise objectives, defining a story, skills like ARP-Posioning, duplication in DNS entries establishing points weight-age and creating a virtual and firewall/routers assessment as seen in figure 1. environment for the cyber security exercise. These cyber-security exercises are usually conducted within hours and days but the time required to pre- • Dry run The dry run is the testing of the devel- pare these cyber-security exercises often spans up to oped virtual environment according to exercise ob- jective by cyber-security experts. This process also Copyright c by the papers authors. Copying permitted takes long time due to the changes and adjustments for private and academic purposes. In: Joseph Collins, required for the cyber-security exercise. Prithviraj Dasgupta, Ranjeev Mittu (eds.): Proceedings of the AAAI Fall 2018 Symposium on Adversary-Aware Learn- • Execution This is the phase where the actual cyber- ing Techniques and Trends in Cybersecurity, Arlington, VA, security exercise takes place. Teams of attackers and USA, 18-19 October, 2018, published at http://ceur-ws.org defenders try to achieve the set of defined objectives. Based upon the complexity of cyber-security exercise Michigan cyber-range (Jones et al. 2015). SVED on the it can take hours to days. other hand utilizes freely available exploit tools such as • Evaluation At this phase teams performance is as- metasploit and nmap and automate their operations to sessed according to the level of exercise objective execute red team activities in a cyber security exercise. completion. Feedback from participants is collected SVED is deployed at CRATE cyber range (Sommes- for future exercises. This phase usually takes few tad 2015). In term of cyber-defense process execution hours for its completion. it is identified that in a cyber-security exercise skilled human professionals are required to conduct the ex- • Repetition The whole process is repeated for a set ercise. Most of the cyber-defense research is focused of new teams utilizing the lessons learned from the on antivirus, antimalware, firewall and SIEM develop- previous exercises and making necessary changes. ment, which left a lot of room for improvement in cyber- Conducting cyber security exercises in the described defense process execution without human involvement manner is a length, tedious and error-prone pro- in a cyber-security exercise. cess (Beuran et al. 2018). Therefore, it is the posi- In term of evaluation phasen in most cyber-security tion put forward that cyber-security exercises are a good exercises the theme of the exercise is CTF (Capture tool for cyber-security skill development but the ineffi- the flag competition). As the name suggests flags are ciencies in cyber-security exercise development and ex- used for point scoring and evaluation purposes. Flags ecution life cycle limits its ability to be widely used for contain some value of a random length when submitted cyber-security skill development. to exercise or competition management systems points will be awarded. Based upon the number of points SURVEY OF THE LITERATURE at the end of cyber security exercise or CTF competi- Researchers have been trying to reduce the inefficiencies tion, teams are evaluated. But the flag based evaluation present in conducting cyber-security exercise. In term mechanism is not ideal for overall performance analysis of environment preparation phase of cyber-security ex- of individuals and teams. Flags only indicate that they ercise life cycle most of the current research is fo- either successful or not in completing a task, flags dont cused on reducing the time required for the preparation indicate at which approach they use or at which stage of virtual environment. Researchers developed multi- they feel difficult in completing the task. To tackle ple solutions for this problem two of them are Tele- this problem KYPO (Čeleda et al. 2015) cyber range lab (Willems and Meinel 2012) and SecGen (Schreuders implemented an evaluation mechanism that is depen- et al. 2017) (security scenario generator). In TeleLab dent upon event log monitoring. Event logs contains the researchers created multiple templates of virtual specific information about the activities that are being environment and developed an environment definition performed on a system. Based upon this information language, through which existing template are modi- automatic evaluation is performed. fied automatically to create new virtual environments for cyber security exercises. In SecGen researchers take ANALYSIS AND DISCUSSION the approach of TeleLab a bit further. Instead of defin- The literature contains interesting solutions for the re- ing the detailed environment schema using an environ- duction of inefficiencies in cyber-security exercise de- ment definition language, SecGen takes the environ- velopment and execution life cycle. But these solutions ment requirement i.e. number of machine, number of have their cons as well. If we consider the autonomous vulnerabilities, type of vulnerabilities etc. as input and generation of experimental environment by TeleLab and randomly generate a virtual environment through the SecGen, we will notice that the environment which is combination of existing virtual environment templates. generated is based upon an environment that is already In the dry-run phase of cyber-security exercise recent available and if a participant already participated in an studies (Ošlejšek et al. 2018) has shown that this phase environment that is used for the creation of the environ- has a lot of room for improvement. It is identified that ment then the participants will have unfair advantage. team of human attacker and defenders does manual ver- The autonomous attack execution in the cyber- ification of the developed environment. That makes the security exercise by SC2RAM and SVED gives a ca- process quite inefficient. pability to the team of defenders to practice their skills In execution phase multiple solutions in the litera- without the availability of an actual attacker. But these ture are available for the execution of cyber-attacks tools are currently at an initial phase of their testing and defense in cyber-security exercise execution. Two and have only basic capabilities. That makes them un- of the attack execution tools are Simulated Cognitive suitable for realistic training. Cyber Red-team Attack Agent (SC2RAM) (Jones et The scoring mechanism in KYPO cyber range is a al. 2015) and Scanning, Vulnerabilities, Exploits and very good approach for automatic evaluation of a par- Detection tool (SVED) (Holm and Sommestad 2016). ticipants performances in a cyber security-exercsie by SC2RAM is developed to mimic the red team execution monitoring the event logs created by the participants steps in a cyber security-exercise. It can perform ba- activity. However, this approach can only give a holis- sic DoS (Denialof-Service) attack on a given network. tic view of participant performance, which is only good It is still at prototyping stage and is being tested at for calculating the overall performance of a participant, not the performance of a participant at specific phase life cycle .This will (1) reduce the cost and time re- of the cyber-security exercise. quire for conducting cyber-security exercise, (2) provide better training by always-available autonomous adver- POTENTIAL SOLUTION saries, and (3) make cyber-exercises computationally repeatable for conducting systematic training. Research is being carried out to address the issues present in conducting operation based cyber-security exercises. Researchers in (Jones et al. 2015) presented References a novel technique to model and execute an active op- Beuran, R.; Tang, D.; Pham, C.; Chinen, K.-i.; Tan, position in a cyber-security exercise. The researchers Y.; and Shinoda, Y. 2018. Integrated framework for discussed the missing element in the exercise environ- hands-on cybersecurity training: Cytrone. Computers ment that is active opposition. The researchers argued & Security. that:The environment may have static defenses, such Čeleda, P.; Čegan, J.; Vykopal, J.; and Tovarňák, as access control or firewalls, or a fixed set of intrusion D. 2015. Kypo–a platform for cyber defence exer- methods to defend against, but it typically lacks any ac- cises. M&S Support to Operational Tasks Including War tive opposition that might adapt defensive or offensive Gaming, Logistics, Cyber Defence. NATO Science and actions (e.g., monitor logs, blocked connections, exploit Technology Organization. switching or information gathering) Furnell, S.; Fischer, P.; and Finch, A. 2017. Can’t get The researchers presented techniques to model cyber- the staff? the growing need for cyber-security skills. attack/defense adversaries and highlighted possible ap- Computer Fraud & Security 2017(2):5–10. proaches that can be used in the implementation of Holm, H., and Sommestad, T. 2016. Sved: Scan- such adversaries. Based upon this research, a tool is ning, vulnerabilities, exploits and detection. In Mil- developed for autonomous execution of highly skilled itary Communications Conference, MILCOM 2016- red-team attackers SC2RAM: A Deployable Cognitive 2016 IEEE, 976–981. IEEE. Model of a Cyber Attacker (Jones et al. 2015). This tool can train blue teamers to tackle cyber-security chal- Jones, R. M.; OGrady, R.; Nicholson, D.; Hoffman, lenges and can configure and test defensive systems. R.; Bunch, L.; Bradshaw, J.; and Bolton, A. 2015. SC2RAM is deployed at Michigan cyber-range to per- Modeling and integrating cognitive agents within the form basic cyber-attack simulation, as it is still at pro- emerging cyber domain. In Proceedings of the Inter- totype stage. On the other hand tools that mimic blue service/Industry Training, Simulation, and Education teams actions in a security exercise is still need to be Conference (I/ITSEC), volume 20. Citeseer. implemented (Jones et al. 2015). We are planning to Moore, E.; Fulton, S.; and Likarish, D. 2017. Eval- model the roles of white, blue and red teamers with uating a multi agency cyber security training program respect to each other for the development of a cyber- using pre-post event assessment and longitudinal analy- security exercise platform that can assist execution of sis. In IFIP World Conference on Information Security cyber-security exercises in a autonomous manner by Education, 147–156. Springer. autonomously preparing the exercise environment and Ošlejšek, R.; Vykopal, J.; Burská, K.; and Rusňák, V. generating autonomous adversaries according to the ex- 2018. Evaluation of cyber defense exercises using visual ercise environment. This will effectively remove the analytics process. need of human adversaries and support staff required Schreuders, Z. C.; Shaw, T.; Ravichandran, G.; Keigh- for conducting a cyber-security exercise. By reducing ley, J.; Ordean, M.; et al. 2017. Security scenario gen- these inefficiencies cyber-security exercises can be con- erator (secgen): A framework for generating randomly ducted regularly at a wider scale, which will help in vulnerable rich-scenario vms for learning computer se- reducing the cyber-security skill gap currently present curity and hosting ctf events. In USENIX. USENIX in industry. Association. Sommestad, T. 2015. Experimentation on operational CONCLUSIONS cyber security in crate. NATO STO-MP-IST-133 Spe- From the above discussion it can be observed that mul- cialist Meeting, Copenhagen, Denmark. tiple phases involved in cyber-security exercise devel- Vykopal, J.; Vizváry, M.; Oslejsek, R.; Celeda, P.; and opment and execution can be automated to reduce cost Tovarnak, D. 2017. Lessons learned from complex and time required for conducting cyber-security exer- hands-on defence exercises in a cyber range. In Fron- cises in an efficient manner. As it was suggested ear- tiers in Education Conference (FIE), 1–8. IEEE. lier inefficiencies in cyber-security exercise development and execution life cycle limit its ability to be widely used Willems, C., and Meinel, C. 2012. Online assessment for cyber-security skill development. We can conclude for hands-on cyber security training in a virtual lab. In that the roles of white, blue and red teamer in a cyber- Global Engineering Education Conference (EDUCON), security exercise need to be executed autonomously, 2012 IEEE, 1–10. IEEE. which will increase the efficiency of preparation, exe- cution and evaluation phases in cyber-security exercise