Cyber ranges are becoming a widely used alternative to teach trainees security through practice on realistic systems. They support evaluation and awareness through dashboards that display how many training objectives have been achieved in a cyber exercise. In our previous paper [1] we have outlined all the limitations of standard dashboards and proposed a new framework for modeling and assessing trainee activity through trainee graphs, reference graphs and scoring functions. In particular, we have introduced a scoring function based on the symmetric diference between a trainee and a reference graph, which allows to pinpoint quite efectively the ineficiencies of a trainee in an exercise. In this paper, we show how that model, while working well for small and coherent exercises, poses several problems when applied to a larger labs made of several, heterogeneous challenges. The accuracy of the symmetrical diference rapidly drops down as the number of nodes and edges in the reference graph increases, thus making it impossible to use it in large environments. Furthermore, there might be edge cases where trainees that do not complete an exercise obtain a higher score than those who do. This happens because the symmetric diference turns out to be higher if the trainee advances exploring fewer nodes of the reference graph (which is the case of a skilled attacker). To address these problems, we aim at reducing the complexity of the graphs fed to the scores. We improve the older model by representing an exercise as a set of smaller local graphs (each one for a coherent, intermediate challenge which can be assessed with a specific local score) and a global graph (representing the interconnection between intermediate challenges, that can be assessed with a specific global score). The main benefits of introducing global and local graphs using global and local progress are twofold: (a) having smaller graphs, scores related to precision (symmetric diferences) are more performant; (b) we can assign diferent scores to diferent parts of the exercise, which is crucial in heterogeneous engagements. We have implemented a Python-based simulator that generates random exercises and compares the performance of the previous and proposed trainee models under specific scores. Our results empirically show that, on average, the original model fails to scale to sizes in the order of tens of vertices and or edges, while the new is able to preserve precise scores locally and better track overall progress.
General consensus states that security is best learned by practice on realistic systems including
workstations and servers, IoT devices and cyber-physical systems. A cyber range is a
prearranged virtual environment that allows trainees to simulate realistic attack and defense
scenarios on an architecture resembling some original system. Cyber ranges typically support
evaluation and awareness through dashboards that display how many training goals have been
fulfilled. This approach does not allow to identify the reasons behind a bad trainee performance.
To overcome this limitation, Andreolini et al. [
While these scores work well for small and coherent challenge environments (e.g., a single host executing a Web server), several problems arise when trying to apply this approach to a larger environment (e.g, heterogeneous multi-host labs located in diferent subnets). The accuracy of the symmetrical diference decreases abruptly as the number of nodes and edges in the reference graph increases. Furthermore, there might be edge cases where trainees that do not reach the exercise main goal obtain a higher score than those who do. To address these problems, we aim at reducing the complexity of the graphs fed to the scores. We model an exercise as a set of smaller local graphs (representing a coherent, intermediate challenge which can be assessed with a specific local score) and a global graph (representing the interconnection between intermediate challenges, that can be assessed with a specific global score). The main benefits of introducing global and local graphs using global and local progress are twofold: (a) having smaller graphs, scores related to precision (symmetric diferences) are more performant; (b) we can assign diferent scores to diferent parts of the exercise, which is crucial in heterogeneous engagements.
We have implemented a Python-based simulator that generates random exercises and compares the performance of the previous and proposed trainee models under specific scores. Our results empirically show that, on average, the original model fails to scale to sizes in the order of tens of vertices and or edges, while the new is able to preserve precise scores locally and better track overall progress.
The paper is organized as follows. Section 2 briefly recalls the original scoring model
introduced in [
In this section we briefly recall the original scoring model introduced in [
a2 a1 a3
S1 S2 S3 a4 a5 a6
S4 a7
S5
End graph tracks the actions performed by a trainee during an exercise. It is built automatically ofline (at the end of an exercise, for a post-mortem performance analysis) or online (during an engagement, to track live trainee progress on a dashboard). The build process uses a reference graph and a set of metrics collected on the game network (e.g., command history and Web browsing history) during the exercise. These metrics allow to built an event timeline and to match it with the intermediate states of a reference graph. Whenever an event in the timeline matches an edge (, ) in the reference graph, the trainee graph is updated as follows: vertex is added to the trainee graph; vertex is located in the trainee graph; an edge (, ) is added to the trainee graph. Figure 2 shows the incremental update of a trainee graph with vertex 2 and edge 2. Matching of timeline events with trainee actions is done in ordered fashion on all nodes of the reference graph. If, on the other hand, a timeline event cannot be Lab Network
M1 M2 ... Mn Metrics
Framework
Events tiEmveelinnte e1 e2 ... en
Match Update
S2
e4 End
Trainee graph e1 Start e2 S1
S2 matched against any edge in the reference graph, the trainee graph is updated as follows: the current vertex in the trainee graph is located; the next expected vertex in the reference graph is identified (such a vertex always exists, be it “start” if the trainee hasn’t yet followed the recommended solution, or an intermediate one if the trainee has followed some steps of the exercise); the label of the vertex is identified; a new dummy vertex is added to the trainee graph with label _ (if it already exists, omit vertex insertion); an edge (, ) is added with label set to . Figure 3 shows the insertion of a dummy node in a trainee graph with event .
Trainee graph e1 Start e S1
S2_err
Reference graph e1 Start e2
S1 S2 e3 S3 e4
End
A score is a function : → R that takes as input a set of oriented graphs (including at
least one trainee and a reference graph), and outputs a real number in a specified interval
[, ] (e.g., [
Score 0. Let be a trainee graph and a reference graph. Let and be the length of the shortest path from the starts vertex to the end vertex in the trainee and reference graph, respectively. We define 0 in the following equation: 0 = {︃ 0 if the trainee completes the exercise otherwise The rationale is as follows. If the trainee completes the exercise, the fraction rewards shorter exploitation paths ( is lower). An unsolved exercise is not rewarded.
Score 1. Let = (, ) be a trainee graph and = (, ) a reference graph. We
consider the symmetric diference of two sets , : △ = ( ∖ ) ∪ ( ∖ ). In other
words, △ contains all members of not in and all members of not in . We use two
interesting properties of symmetric diference: (a) if two sets and coincide, △ = ∅; (b)
if two sets and are disjoint, △ = ∪ . We define the eficacy as following:
= 1 −
|△|
| ∪ |
If the trainee and reference graph coincide, we have △ = ∅, thus = 1; the trainee
has followed exactly the sequence of intermediate states modeled by the reference graph, and
obtains the highest possible score. On the other hand, if the trainee and reference graph are
completely disjoint, we have △ = ∪ , thus = 0; the trainee hasn’t reached one
single intermediate state of those in the reference graph, and obtains the lowest possible score.
Intermediate performance produces scores in the [
Score 1 is defined as a linear combination of and : 1(, ) = + (1 − ) In this paper, we choose = 0.5, but the score may be tuned to weigh more eficacy or eficiency, according to the specific learning objective. This score rewards trainees who are able to reach the final goal of an exercise following the path defined by the reference graph. Trainees who could not reach the goal or make many mistakes are penalized with a low score. This is a good candidate for teaching exercises, since it tracks trainee actions during the exercise, taking into account the ability to follow the taught path to reach a goal.
While the original scoring model introduced in [
To address these limitations we change the scoring model with the aim to reduce the complexity of the graphs fed to the scoring functions, as shown in the following section.
Instead of modeling an exercise as a single, potentially large graph we split it as a set of smaller, local graphs ( ∈ [1, ]) representing a single intermediate step. Figure 4 shows a possible local reference graph with local start, end and intermediate vertices ,, , and , respectively ( ∈ [1, ], ∈ [0, 7]). Local trainee graphs are built exactly as in the older model. Each local graph is assigned a scoring function : → R ( ∈ [1, ]) that assesses trainee performance in that specific intermediate step.
LSi,1
LSi,2 LSi,start
LSi,0
LSi,3
LSi,4
LSi,5
LSi,6
LSi,7
LSi,end
Similarly, we create a global graph that models the interconnection between the intermediate steps. Figure 5 shows a possible global graph with global start, end and intermediate vertices , and respectively ( ∈ [0, ]). Global trainee graphs are built exactly as in the older model by observing transition events from an intermediate challenge to another. The global graph is assigned a scoring function : → R that assesses trainee performance in advancing through the intermediate challenges.
GSstart
GS0
GS1
GS2
In principle there is no impediment to assigning multiple scores to a specific graph (e.g., to simultaneously capture multiple skills such as speed and precision); however, for reasons of simplicity in this paper we will use only one score per graph. At the end of an exercise, the instructor has available: • a set of + 1 graphs (, 1, 2, . . . , ) that track out the overall advancement of a trainee through the intermediate challenges and in every single challenge; • a set of + 1 scores (, 1, 2, . . . , ) that quantify the aforementioned advancement and allow to compare it against the performance of other trainees (or even the same trainees over several, repeated exercises).
Aggregating these scores to produce a leaderboard is a very interesting topic that is out of scope for the current paper. Here, we show that the scores produced by the new model are more consistent than those of the old one in large graphs.
In Section 4.1 we briefly discuss the simulator, the hardware/software testbed and the simulation scenarios carried out. In Section 4 we compare the performance of the 1 score in the old and new model under diferent simulation scenarios.
We have implemented a simulator that models an exercise based on a multitude of hosts
organized in multiple networks. Each host exhibits a specific challenge which the trainee is
supposed to solve. The trainee is also supposed to advance through the lab in a specific sequence.
This is a quite popular model in the cyber exercise world (e.g. HackTheBox Pro Labs [
After setting up the graphs, the simulator performs random walks over the global graph and, for each node of the global graph, performs random walks into the associated local graph. To speed up simulations, local random walks are performed with two diferent error rates: 0.95 (related to a beginner) and 0.0 (related to an expert). The goal is twofold: (a) simulate the progress of a trainee through diferent hosts in the lab; (b) simulate the progress of both a high skilled and a rookie trainee into a specific challenge. At every step of the global and local random walks we compute the appropriate global and local scoring functions. Finally, at the end of an exercise the simulator saves the test configuration ( , the local and global reference and trainee graphs, the global and local scores) into separate files.
The simulator has been developed in Python 3.11.7 and uses the networkx package to handle large graphs. Simulations have been run in a host with an Intel Core 12700H CPU, 16 GB RAM and 1 TB SSD, running the ArchLinux GNU/Linux distribution.
Figure 6 shows the evaluation of scores 0 and 1 under diferent simulation scenarios. Labels have the form score/model/stats, where “score” is either the 0 or 1 score, “model” refers to the old-style or new-style trainee graphs (local or global in the latter case) and “stats” is either the minimum or maximum score value observed during a specific simulation. Let’s focus on the 1/old/max and 1/old/min labels that represent the maximum and minimum values of 1 computed with the old model in exercises of increasing complexity in terms of vertices. As can be seen, the maximum value of the symmetric diference drops with , making 1 unusable starting with approximately = 15 machines. The ratio of vertices in common between a trainee graph and a reference graph tends to drop as the number of vertices and edges increases. Keeping a high 1 score would mean that an attacker exploits every node in exactly every possible way as defined by an instructor, which is very hard and often simply not needed. On the other hand, the minimum value stabilizes to zero for pretty much the whole range of considered hosts; this means that bad performance is consistently evaluated. A very similar behavior is exhibited by the 1 score in the new model (labels 1/new-global/*) when operating on larger and larger global trainee graphs.
Let us now focus on the other 0/new-global/* and 1/new-local/* labels that represent respectively 0 in the new model (applied on a global trainee graph) and 1 in the new model (applied to local trainee graphs). Here, the aforementioned limitations does not apply; both good and bad performance is consistently evaluated for the whole range of considered hosts. In particular, the 0/new-global/max plot is consistently high because 0 rewards very highly a trainee that is able to find the shortest exploitation path (which almost always happens in repeated simulations). Regarding 1/new-local/max, 1 performs extremely well due to the constant and low number of ten nodes in every local graph generated by the simulator.
Table 4.2 shows the values of the 1 score in a pathological simulation representing an exercise run in a lab of = 50 nodes. We show only this value for reasons of space, but we have verified that the problem persists basically for every value of , and is more frequent with increasing values of . The old reference graph has a shortest path to the goal of length 1. Two trainees carry out the exercise, each one exhibiting diferent skills (a beginner with error rate 0.95 and an expert with error rate 0). The expert trainee finds the shortest path and completes the challenge, while the beginner trainee develops an incomplete exploitation path of length 9 and fails to complete the challenge. One would expect a higher score for the expert trainee with respect to the beginner trainee; however, this is not the case. The expert trainee receives a score of 0.035, while the beginner trainee receives a score of 0.07 (exactly doubled). The reason for this anomaly lies again in the way symmetrical graph diference works. The beginner trainee graph shares a longer path with the reference graph than the expert trainee would; this leads to a higher similarity of graphs and a higher 1 score.
Error rate
Previous literature has explored formal representations to articulate an attacker’s maneuvers
in terms of the techniques utilized and vulnerabilities exploited within systems and
configurations. Attack trees [
It is worth pointing out that as the number of vertices and edges in attack trees increases, their complexity escalates rapidly. Specifically, it becomes increasingly resource-intensive to trace all pathways from a leaf node to the root node. In realistic scenarios, the network of nodes and connections often surpasses thousands, where the mere addition of a single node significantly expands the number of arcs and potential attack routes. Moreover, given that the root node symbolizes the ultimate aim of the attack, complex multi-stage attacks may necessitate the utilization of multiple attack trees for accurate representation. Attack graphs [7] represent the infrastructure requiring protection, including network topology, vulnerable assets, and available exploits. They point out the pathways an attacker must traverse to achieve specific objectives. All the aforementioned approaches ofer a static perspective of attacks and mitigation strategies, but they fail to track the progress of an attacker on a live system.
In the nascent stages of scoring systems research for cyber ranges, the main approaches
are focused on signaling goal completion rather than evaluating trainee performance, as in
[8], [9], [10]. To the best of our knowledge, [
In this paper we have discussed our previously published trainee and scoring model [
Future work will be centered on fixing the scoring anomaly produced by 1 in some corner
case simulations and on implementing the new scoring model in the prototype presented in
[
NATO Science and Technology Organization (2015). [9] J. Vykopal, M. Vizváry, R. Oslejsek, P. Celeda, D. Tovarnak, Lessons learned from complex hands-on defence exercises in a cyber range, in: 2017 IEEE Frontiers in Education Conference (FIE), IEEE, 2017, pp. 1–8. [10] M. Carlisle, M. Chiaramonte, D. Caswell, Using ctfs for an undergraduate cyber education, in: 2015 {USENIX} Summit on Gaming, Games, and Gamification in Security Education (3GSE 15), 2015.