Computer network defence is a complicated task that has necessitated a high degree of human involvement. However, with recent advancements in machine learning, fully autonomous network defence is becoming increasingly plausible. This paper introduces an end-to-end methodology for studying attack strategies, designing defence agents and explaining their operation. First, using state diagrams, we visualise adversarial behaviour to gain insight about potential points of intervention and inform the design of our defensive models. We opt to use a set of deep reinforcement learning agents trained on diferent parts of the task and organised in a shallow hierarchy. Our evaluation shows that the resulting design achieves a substantial performance improvement compared to prior work. Finally, to better investigate the decision-making process of our agents, we complete our analysis with a feature ablation and importance study.
Computer network security is characterised by an asymmetry as the defender needs to ensure
constant protection of the network’s components, while the adversary can opportunistically
single-out weak entry points. Such asymmetries have been identified and addressed in many
other areas of cyber security. For example, cryptographic protocols (e.g., TLS) thwart denial
of service attacks by ensuring that the prover commits enough computation cycles before
the verifier does so. In network defence, however, the problem remains open as the task is
complex [
RL, and particularly deep RL (DRL), excels in interactive tasks that cannot easily be solved
using analytical solutions. Human and even super-human levels of performance have been
achieved in a range of complex tasks including classic board games such as chess and Go [
To date however, there has been limited consideration for the explainability of these models. Explainable AI has, in domains such as natural language processing and computer vision [10], proven useful not only for end users but also experts and developers of AI systems. DRL models are particularly challenging to explain because the neural networks which represent their agent policies are not readily understandable by humans. Nonetheless, the ability to explain and understand the actions of an autonomous defensive agent is critical. This work investigates, and answers in the afirmative, whether explainable RL (XRL) models and environments can improve autonomous defensive capabilities and aid in their development.
• We develop methodologies for visualising (i.e., explaining) attacker functionality in the CybORG cyber environment. Our methodology highlights previously undocumented diferences in the adversary models and motivates two new controller architectures with improved classification accuracy. • We present the full details of our new controller and specialised subagent models. We then evaluate them against two classes of adversary in the CybORG environment realising substantial performance improvements. • We perform a feature ablation and importance study to understand the most influential elements in the observation space and explain our model outputs.
In this section we discuss the key RL techniques that are relevant for the rest of the paper.
2.1. Deep RL Algorithms
2.1.1. PPO
Proximal Policy Optimisation (PPO) is an eficient policy gradient method [ 5] for DRL. It has
been shown to outperform other popular algorithms such as A3C [
PPO uses a policy ( ∈ R) with an objective function that is defined by the total reward ( ) = E 0 [∑︀∞
=0 ]. By formulating the objective function in this way actor-critic architectures can be used: the actor selects an action which is evaluated by the critic. The policy gradient is then computed: ▽ ( ) = E 0 [▽ log (, ) ()] (1) () is the advantage of taking action instead of the average action as computed by the policy ( (, ) − ()) [14].
During gradient descent, PPO introduces a clipping function to both prevent reaching local
optima during large updates and avoid smaller updates that significantly increase the length of
training.
2.2. Curious Exploration
Curiosity is a technique that enables agents to explore their environment based on an intrinsic
reward signal not provided by the environment [15]. Such a signal is particularly useful in the
absence of a continual extrinsic reward (e.g., the running score found in some games).Pathak et al.
[15] introduce the Intrinsic Curiosity Module (ICM), a semi-supervised technique in which agents
choose actions based on the uncertainty in the outcome of each action, intrinsically motivating
the exploration of unknown states. ICM also ensures that agents are only incentivised to reach
states that are impacted by their actions, avoiding those which are inherently unpredictable.
2.3. Explainable RL
Explainable RL (XRL), a fledgling sub-field of explainable AI, is the study of tools and methods
which enhance human understanding of the actions taken by autonomous agents. A recent
and thorough review of XRL is provided by Heuillet at al. [16] and separately by Puiutta and
Veith [17]. XRL methods are commonly divided between those which are intrinsic, sometimes
called transparent, and those which are post-hoc. Intrinsic XRL models are inherently
interpretable and ofer explainability at the time of training. In contrast, post-hoc explainability
occurs after training; often by creating a second, simpler model to provide explanations. In DRL,
learned policies are represented by neural networks making them dificult to interpret. Post-hoc
explainability allows the performance advantages of DRL [
We use the CybORG environment [18] which simulates the computer network of a manufacturing plant, as shown in Figure 1. The network consists of five user hosts (Subnet 1), three enterprise servers (Subnet 21), three operational hosts and the operational server (Subnet 3). Each host exposes a number of network services that other hosts can connect to, and which may have exploitable vulnerabilities. However, due to the network’s firewalls hosts in Subnet 1 cannot directly connect to machines in Subnet 3, and the operational server is accessible only through the operational hosts. The liveness of the operational server has a direct impact on the manufacturing and is considered critical. CybORG assumes two players, a defender and an adversary, who interact with the turn based environment using the actions available to them.
User Hosts Subnet 1
Subnet 2
Subnet 3
Defensive Agent
A common drawback of simulated environments in RL is the reality gap which causes agents not to generalise suficiently when moved from the simulation (i.e., training) to reality (i.e., evaluation). This is due to the simulation not adequately matching reality (e.g., in robotics). To address this, CybORG provides a network emulator that runs on Amazon Web Services (AWS). The combination of simulation and emulation ensure that the reality gap is minimised, with the actions available and their efect on the environment consistent across both [18].
The CybORG environment is host to the ‘Cyber Autonomy Gym for Experimentation’ (CAGE)
challenge [
2Honeypot refers to a decoy system or service that lures attackers by appearing to sufer from known security vulnerabilities. Honeypots are used to detect malicious actors and study their behaviour.
As in a real network defence situation, neither the defender nor the adversary is omniscient. Neither agent knows the state of the network or the other’s position with absolute certainty. In addition, the outcomes of actions are stochastic. For example, from the defender’s perspective, when an exploit fails it is not possible to precisely determine which exploit was attempted. This can be crucial information in the instance that an adversary favours a specific exploit strategy. A better informed defender could strategically place decoys on the targeted service to frustrate and evade further attempts more efectively. 3.3. Reward Function Most games include a scoring function that quantifies the performance of the player. Similarly, CybORG uses a reward function that rewards the adversary and penalises the defender for every compromised or impacted network host. The reward function is as follows: on each turn, for every host on which the adversary has admin access, the defender receives a reward of -0.1 and for every server the reward is -1. There is a -10 reward for disruption on the operational server and a -1 reward when any device is ‘restored’. In the context of RL, the negative reward for the defensive agent incentivises the agent to take actions that minimise the efect of the adversary. 3.4. Adversaries The environment includes two adversaries: the BLineAgent that has prior knowledge (i.e., full knowledge of the network’s structure but not its current state), and the MeanderAgent which does not have any prior information. Both agents share the same objective, to reach the operational server and, after escalating their privileges, disrupt its services (i.e., impact its liveness). Due to prior knowledge, the BLineAgent follows an optimal exploitation trajectory to the operational server. In contrast, the MeanderAgent breadth-wise scans the network for vulnerable hosts and gradually traverses the subnets. To prevent trivial defence strategies, the adversary is given user access on a predetermined host (in Subnet 1) that cannot be ‘restored’ to a benign state by the defender.
Controller
MeanderAgent Defence
BLineAgent Defence
The models that we train have a similar basic structure to those described in [9] that were trained for CAGE I. In particular, we focus our eforts on training a hierarchy of specialised defensive agents using DRL. These agents feature a controller agent that, at each time step, chooses a subagent to perform the action. Each subagent is trained against a specific adversarial strategy.
As described in Section 3.4, the environment includes two adversaries. The hierarchical architecture was developed specifically to exploit this. The model supports two expert subagents that, through the controller, are ‘consulted’ over the course of an episode (Figure 2). This avoids the performance limitations of a single, more general agent. Given the diferences in the two adversaries, each subagent requires a diferent neural architecture for best performance. These are described below. 4.1. MeanderAgent Defence Our MeanderAgent defensive subagent was trained using the PPO algorithm and utilises a comparatively deeper neural network including three hidden layers with widths 256, 256, and 52. Full details of the hyperparameters used can be found in Appendix A.
Notably, curiosity did not improve the performance. Since the MeanderAgent is explicitly designed to explore the network during its attack, the opposing defender is also be forced to explore more broadly and to employ a wider range of strategies during training. As such, it learns suficiently general strategies without the need for curiosity. 4.2. BLineAgent Defence In contrast to the MeanderAgent, the BLineAgent follows a near-optimal path through the network. The BLineAgent defence, therefore, is at much greater risk of overfitting during training. As a result, we found that when training defensive agents against the BLineAgent, it was beneficial to include the curiosity mechanism. In this paper we consider two subagents for BLineAgent defence: an Action Knowledge (AK) subagent, and a State Representation (SR) subagent. Both are trained using PPO with curiosity but make diferent modifications to the state space.
The AK subagent modifies each observation by appending a single bit indicating the success of the previous action. We find that this gives the subagent a better understanding of the defensive process and results in an improvement in performance.
Secondly the SR agent is identical to the AK subagent, but receives observations of 27 floats as opposed to 53 bits. In this state space, each host has two floats to represent the features of activity and compromise. The additional float indicates whether the previous action succeeded. Although the mean episode reward is comparable to the AK agent’s mean reward, we see a notable decrease in variance.
(a) (b)
The behaviour of the adversaries is dependent on the network topology and the choice of defensive actions. In addition, there is stochasticity in both the choice and outcome of actions across all of these components. Explaining adversarial behaviour proved essential in developing efective defensive models. To better understand each adversary we, at each time step, record the choice of action, outcome and the resulting state transition. For consistency across multiple episodes we resolve IP ranges and addresses to subnets and hostnames, respectively. We observe that the connectivity (i.e., the edges) of the resulting graph provides a clear signal for diferentiating the two adversaries. Figure 3 shows a subset of the observations, recorded during the first four steps of adversarial behaviour, in which the BLineAgent and MeanderAgent can be seen adopting a depth-first and breadth-first approach to attacking the network, respectively. In Section 6 we present two methods which make use of this observation to more accurately determine the class of adversarial threat than in prior work [9]. In Appendix B we include the fully extracted adversary specifications generated by our methodology.
In order to improve the performance of our defensive capability we explore the use of alternative controller models. We introduce two new types of controller for this task, one heuristic and another bandit-based. 6.1. Bandit Controller Model We employ a bandit controller that is based on the multi-armed bandit architecture. The task is to determine which of the adversaries is currently attacking the network, based on the sequence of observations. However, using a bandit or bandit-like approach comes with several challenges in this setting.
In the traditional multi-armed bandit there is no notion of state: an agent takes actions and then observes the reward. However, in the CybORG environment a unique observation cannot be used to determine the current adversary. Thus sequences of observations need to be observed and, due to the stochasticity, there are multiple sequences that can be observed over a given number of timesteps. A single bandit predicting the adversary will do no better than 50%.
This is analogous to the traditional multi-armed bandit setting. Consider the task of determining which of two slot machines has the higher payout in a casino (A): the task is trivial after several attempts. Now consider a second identical casino (B) where the payout of the machines is flipped. Again, we can find the better machine in B after some error. Finally, consider being randomly placed in A or B and having only one attempt to select the slot machine with the highest payout. As we do not know which casino we are in (as everything is identical), the best possible guess rate is 50%.
We are able to solve this problem by abstracting the observations (which casino you are in) from the bandit. In this way we define bandits, one for each of the observations. As such the observation is unique to the bandit predicting the adversary. While this could also be solved by a logistic regression model, the Bandit Controller is able to learn with fewer samples, also being able to determine new adversary behaviours and learn to predict them in an online fashion.
The bandit learning algorithm, shown in Algorithm 1, allows the bandit controller to track the states that it has previously seen, creating a new bandit for each newly seen state. Each of these bandits is initialised with values for each of the actions ∈ {0, 1, 2}, where these correspond to the MeanderAgent, BLineAgent, and no adversary. The values are updated using reward and the number of times that prediction has been selected, (). We train the bandit controller for 15,000 timesteps, using = 0.01.
The Bandit Controller has a state diferent to that of its subagents. Its state is a sliding window of the last four timesteps from the CybORG environment. As we can see from Figure 3b, the minimum number of actions before an adversary has user privilege (and the first unambiguous instance of malicious behaviour) is three. A defensive agent can observe this on the fourth timestep, hence a prediction from the bandit controller only needs to happen once per episode. Finally, we use a simple reward function of +1 for a correct prediction, and -1 for an incorrect prediction. 6.2. Heuristic Controller Model We also construct a heuristic for predicting the adversary. This approach is possible as we are able to observe the patterns that the adversaries display in a controlled version of the CybORG environment. As we can see in Figures 3b and 3a, the BLineAgent and MeanderAgent have fundamentally diferent strategies in the first four moves they make. Using this privileged view of the adversarial behaviour allows for a manual and formal definition of the behaviour, as defined in Heuristic 1. As in the Bandit Controller we use this heuristic once per episode, on the fourth timestep, to determine which adversary is attacking the network. Algorithm 1 Bandit Controller Learning Algorithm.
Initialise the known states, Initialise set of bandits, Initialise for a = 1 to k: 0.() ← 0 // Initialise Q values and action counter for the first bandit 0. () ← 0 Predict(): if ̸∈ : ← Initialise ←
{︃(.()) with probability 1 − ← with probability ← _() . () ← . () + 1 .() ← .() + (1) [︀ − .()] Heuristic 1. The scanning of two diferent hosts on the network within the first four timesteps indicates the presence of the MeanderAgent adversary. Otherwise, this is either the BLineAgent adversary or the User agent.
In this section we evaluate the performance of our specialist subagents against the two adversaries. We further investigate the performance of the controller models. Finally, we evaluate the full defensive model capable of defending against either adversary. We use the model described in prior work [9] as a baseline performance measure (baseline for brevity), as this has been established as state-of-the-art and achieved the best score in CAGE I. Because the scoring function assigns only penalty points (i.e., 0 is the theoretically maximum score), all the reported rewards are negative. 7.1. Specialised Sub Agents
Figure 4 shows the average reward of each defensive subagent as trained against the BLineAgent (left column), and the MeanderAgent (right column). The methods of AK and SR achieve peak rewards against the BLineAgent of -12.227 and -11.465 respectively, both of which are an improvement over the baseline [9] PPO with curiosity based model, which achieves -13.475. Furthermore, removing curiosity negatively impacts the reward against the BLineAgent, as shown clearly in the max reward plot of Figure 4(c)
Base_Bline PPO_Bline AK_Bline SR_Bline Base_Bline PPO_Bline AK_Bline SR_Bline Base_Bline PPO_Bline AK_Bline SR_Bline Base_Bline PPO_bline AK_Bline SR_Bline −10 −20 rd−30 a w eR−40 −50 −60 −70 10 0 −10 rd−20 a w eR−30 −40 −50 −60 −100 −200 rd−300 a w eR−400 −500 −600 −700 −10 −20 −30 rad−40 w eR−50 −60 −70 −80 −90 (g) Figure 4: Mean, maximum and minimum reward of blue subagents against the BLineAgent (left) and the RedMeanerAgent (right) over 10 million timesteps.
6M Timesteps (b)
6M Timesteps (d)
6M Timesteps (f )
6M Timesteps (h)
Base_Meander PPO_Meander AK_Meander SR_Meander Base_Meander PPO_Meander AK_Meander SR_Meander Base_Meander PPO_Meander AK_Meander
SR_Meander Base_Meander PPO_Meander AK_meander SR_meander 0M 2M 4M 8M 10M 0M 2M 4M 8M 10M Max Reward Against BLine Adversary
Max Reward Against Meander Adversary 0M 2M 4M 8M 10M 0M 2M 4M 8M 10M Min Reward Against BLine Adversary
Min Reward Against Meander Ad ersary 0
0 0M 2M 4M 8M 10M 0M 2M 4M 8M 10M Mean Reward with Standard Deviation Against BLine
Mean Reward with Standard Deviation Against Meander 0 0M 2M 4M 8M
10M 6M Timesteps 0M 2M 4M 8M 10M MeanderAgent PPO baseline AK SR baseline
BLineAgent
MeanderAgent
Mean Mean Reward
The diference in mean reward is explained by the maximum and minimum rewards. All models apart from PPO experience a first plateau in maximum reward of -9 and then step up to a second plateau of around -1. The SR agent finds the optimal policy earlier than the AK agent during training. In addition, the minimum rewards of the baseline and PPO model have greater variance than the SR and AK agents, and the AK agent has a marginally higher probability to score very poorly (i.e., below -300). Earlier optimal policy convergence and smaller policy variability makes the SR agent the best model against BLineAgent agent. This corroborates the standard deviation graph, and 1,000-episode evaluation results; in Table 1, where the SR agent displays less negative reward and with a standard deviation that is only a fifth of the AK agent.
Against the meander attacker the PPO and SR agents outperform the baseline (-24.384) with best mean rewards of -17.065 and -19.959 during training. Figure 4 shows the advantage of using a PPO 3-layer architecture which results in higher min and max rewards with reduced variance.
Here we evaluate the performance of our defensive subagents against their separate adversaries. We select the best performing agents from training for evaluation evaluate: PPO defence for the RedMeander and both the AK and SR defence for the BLineAgent. We evaluate each for 1,000 episodes of 100 steps and summarise our results in Table 1. For completeness, we also cross-evaluate our agents against the adversary not seen during training.
Against the RedMeander adversary, PPO defence outperforms the baseline against both adversaries resulting in a mean score of -21.3 (improvement by a factor of 3.6) and a reduction in standard deviation by a factor of more than 9. This highlights the advantage of the increased depth of the neural network over the baseline.
Against the BLineAgent adversary, we see that the SR agent is able to achieve a 1.5 times greater reward, with 4.89 times lower standard deviation. However, this comes at the cost of generality. A trend in all of the subagents is that when defending against previously unseen adversaries, the performance is significantly diminished. PPO with curiosity (4 steps) PPO with curiosity (100 steps) Heuristic Bandit BLineAgent
RedMeander 7.2. Controller Models As seen in Section 7.1.2, the defensive subagents do not generalise well beyond the adversaries that they are trained against.
To address this, Sections 6.2 and 6.1 introduce two new controller architectures: Heuristic and Bandit. Here we evaluate the ability to correctly predict the adversary within the first four timesteps of an episode (as our controllers predict the adversary on the fourth timestep). For each episode, we randomly sampled one of the two red adversaries (i.e., 50% probability of selecting BLineAgent).
Table 2 shows that the baseline model has strong biases on selecting the BLineAgent agent. To investigate further, we let the baseline agent make predictions on each timestep until the end of the episode (c.f. only guessing after the 4th timestep). As seen in Table 2, the repeated guesses significantly reduced bias but accuracy remained low.
In contrast, neither our bandit or heuristic controller exhibit this bias and can perfectly predict the correct attacker type. 7.3. Hierarchical Defensive Model Here we evaluate the complete defensive model. Table 3 reports the mean and standard deviation for the ‘best pair’ combinations of subagents as determined by our evaluation in Section 7.1 ( i.e., PPO for MeanderAgent, and AK or SR for BLineAgent).
We observe that the subagents play a significant role in the improvement over the baseline. Over episodes of 100 timesteps, we are able to improve the result by at least 30% for the BLineAgent and 170% for MeanderAgent.
The lowest reward values are split evenly between the Heuristic and Bandit controllers. These models outperform the PPO controller models regardless of the subagents in four of the six combinations of adversary and episode length.
MeanderAgent performance is improved by 11.7%, which is more significant than BLineAgent (only improved by 1%) when using Bandit or Heuristic controller. Table 1 indicates that models trained with BLineAgent perform poorly on MeanderAgent. This can be explained by the fact that BLineAgent has more information about the network, so its behaviour is more predictable. In contrast, MeanderAgent’s actions have more randomness. Bandit Heuristic Baseline (PPO Controller) Baseline (PPO Controller)
Subagents PPO + AK PPO + SR PPO + AK PPO + SR PPO + AK PPO + SR Baseline (PPO subagents)
It is critically important that human operators can understand the decisions made by autonomous agents. Using post-hoc XRL techniques, we determine whether our defensive agents are truly defending the network as their primary objective or as a side efect of an unintended objective. This is common in RL where agents may manipulate improperly specified reward mechanics to maximise their score in unintended ways. 8.1. Ablation Study To understand which of the features in the observation space influence the agents decision making we perform an ablation study over knowledge of: 1) the success or failure of the previous action (hence referred to as previous action), 2) the adversary’s access onto a host (hence referred to as adversary access), and 3) whether an adversary has scanned a host (hence referred to as adversary scan).
The ablation results in Figure 5 show the AK and SR agents against the BLineAgent in 5a and 5b, and the PPO agent against the MeanderAgent3 in 5c. Figure 5a indicates that the AK agent’s performance is greatly afected by ‘adversary access’. While comparatively little impact seems to derive from the ablation of ‘adversary scan’ and ‘previous action’ there is some variance and the rewards fall to -812 and -539, respectively. Interestingly, the SR defensive agent is greatly afected by the ablation of the ‘adversary access’ and ‘adversary scan’, with the distribution of rewards being more negative in both cases. This is especially apparent in the case of ‘adversary scan’. Previous action has less of an efect in both AK and SR, yet still reduces the mean reward to -30.42 (a factor of 2) and -40.6 (a factor of 3), respectively. However, AK has some outlier scores that result in a minimum reward of -987.8. For PPO against MeanderAgent, Figure 5c shows that ablation of ‘adversary access’ causes a drastic reduction in reward, bringing the mean value to -781.23. Ablation of ‘adversary scan’ reduces the mean reward to -44.70, a factor of 2.52 more negative than when the observation is included.
3This defensive agent doesn’t use the previous action however we include it for completeness.
AdversaryAccess AdvAerbslaartyioSncan PreviousAction
(c)
−400
−600
−800
−1000
0
−200
ad −400
rew −600
R
−800
−1000
8.2. Feature Importance
To further validate the importance of ‘adversary access’, ‘previous action’, and ‘adversary scan’,
we utilise a well known framework from explainable AI called SHapley Additive exPlanations
(SHAP). This uses an implementation agnostic game theoretic approach to explain the
importance of features in determining outputs. SHAP is able to connect optimal credit allocations
with local explanations to determine SHAPley values. These values provide a way of accurately
distributing the contribution of the individual features within the complete feature space [
Figure 6 shows the SHAPley values for the trained AK and SR subagents against the BLineAgent in 6a and 6b, and PPO against the MeanderAgent in 6c. Each point on these plots is a feature in a specific observation, with the colour representing the value of that feature.
All defensive agents observe the same trend in feature importance regardless of their training adversary: ‘adversary access’ is the most important followed by ‘adversary scan’, a trend that is also observed in Figure 5. Note that the PPO RedMeander defensive model doesn’t use ‘previous action’ and hence is not included in Figure 6c.
We show that ‘adversary access’ is an important part of the observation. This indicates that the defensive agents are aware that they need to remove the attackers from hosts. The importance is also seen in Figure 5 as the most significant shifts in reward distribution occur when ablating ‘adversary access’.
In addition ‘adversary scan’ is of importance to the agents which is clear in 5b as the defensive agent’s performance is significantly impacted in the absence of this information. This correlates with Figure 6b as ‘adversary scan’ has the greatest distribution of any of the SHAP values for the BLineAgent defensive agents. While knowledge of the ‘previous action’ has the lowest feature importance for the agents, we argue that this is still important for these defensive agents, which, with this knowledge, outperform the baseline and PPO-only models in 4. For example, take the case where a defensive agent acts to remove an adversary from a host, if this action fails then the defensive agent will have to adjust its strategy. The importance of this feature can further be seen in Figures 5a and 5b, as ablation of this feature has a non-trivial impact on the performance of the agents. e u l a v e tr u a e F
Adversary Access Adversary Scan Previous Action e u l a v e tr u a e F Lo
Lo
The efectiveness of RL across a range of simulated and abstracted autonomous network defence
scenarios is well established in the literature. Han et al. [
Several alternative network defence simulation environments have been proposed in the
literature. Molina-Markham et al. [
RL has also be applied to several closely related problems. In penetration testing (i.e.,
exploitation which is a subset of the CybORG envionment), Yang and Liu [
The authors would like to acknowledge that research was partially funded by EPSRC grant EP/T51780X/1. [5] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal Policy Optimization
Algorithms, in: arXiv:1707.06347 [cs], 2017. [6] O. et al., Dota 2 with Large Scale Deep Reinforcement Learning, 2019. [7] A. E. Sallab, M. Abdou, E. Perot, S. Yogamani, Deep Reinforcement Learning framework for Autonomous Driving, Electronic Imaging 29 (2017) 70–76. URL: http://arxiv.org/abs/ 1704.02532. doi:10.2352/ISSN.2470-1173.2017.19.AVM-023, arXiv:1704.02532 [cs, stat]. [8] J. Kober, J. A. Bagnell, J. Peters, Reinforcement learning in robotics: A survey, International Journal of Robotics Research 32 (2013) 1238–1274. URL: https://doi.org/10.1177/ 0278364913495721. doi:10.1177/0278364913495721. [9] M. Foley, C. Hicks, K. Highnam, V. Mavroudis, Autonomous Network Defence using Reinforcement Learning, in: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, ASIA CCS ’22, Association for Computing Machinery, New York, NY, USA, 2022, pp. 1252–1254. doi:10.1145/3488932.3527286. [10] J. R. Williford, B. B. May, J. Byrne, Explainable Face Recognition, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision - ECCV 2020, Lecture Notes in Computer Science, Springer International Publishing, Cham, 2020, pp. 248–263. [11] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. Bayen, Y. Wu, The Surprising Efectiveness of PPO in Cooperative, Multi-Agent Games, 2022. URL: http://arxiv.org/abs/2103.01955. doi:10.48550/arXiv.2103.01955, arXiv:2103.01955 [cs]. [12] T. T. Nguyen, V. J. Reddi, Deep Reinforcement Learning for Cyber Security, 2021. [13] X. Wu, W. Guo, H. Wei, X. Xing, Adversarial Policy Training against Deep Reinforcement Learning (2021) 1883–1900. URL: https://www.usenix.org/conference/usenixsecurity21/ presentation/wu-xian. [14] R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, Adaptive computation and machine learning series, 2nd ed., 2018. [15] D. Pathak, P. Agrawal, A. A. Efros, T. Darrell, Curiosity-driven exploration by selfsupervised prediction, in: Proceedings of the 34th International Conference on Machine Learning, ICML’17, 2017. [16] A. Heuillet, F. Couthouis, N. Díaz-Rodríguez, Explainability in deep reinforcement learning, Knowledge-Based Systems (2021). URL: https://www.sciencedirect.com/science/article/pii/ S0950705120308145. [17] E. Puiutta, E. Veith, Explainable Reinforcement Learning: A Survey, in: 4th International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE), 2020. URL: https://hal.inria.fr/hal-03414722. [18] M. Standen, M. Lucas, D. B., T. J. Richer, J. Kim, D. Marriott, Cyborg: A gym for the development of autonomous cyber agents, in: IJCAI-21 1st International Workshop on Adaptive Cyber Defense, 2021. [19] M. Standen, D. Bowman, S. Hoang, T. Richer, M. Lucas, R. Van Tassel, Cyber Autonomy Gym for Experimentation Challenge 1, https://github.com/cage-challenge/cage-challenge-1, 2021. [20] M. Standen, D. Bowman, S. Hoang, T. Richer, M. Lucas, R. Van Tassel, P. Vu, M. Kiely, Cyber autonomy gym for experimentation challenge 2, https://github.com/cage-challenge/ cage-challenge-2, 2022. Created by Maxwell Standen, David Bowman, Son Hoang, Toby
Agent AK SR PPO Bandits
Parameter gamma network layers Curiosity Beta (Curiosity) Eta (Curiosity) Feature Dimension (Curiosity) Learning Rate (Curiosity) Learning Rate gamma network layers Curiosity Beta (Curiosity) Eta (Curiosity) Feature Dimension (Curiosity) Learning Rate (Curiosity) Learning Rate gamma network layers Curiosity Learning Rate epsilon
Value
Optimal, lower and upper bounds of the of the hyperparameters are shown in Table 4. A uniformly sampled grid search was used to determine the optimal values.
Here we provide the full action-outcome transition graphs for the BLineAgent adversary, both with and without the presence of our defensive model. Table 5 provides the definitions of all the acronyms used.
Acronym
Definition
Figure 7: Action-outcome transition graph of the BLineAgent adversary without defensive action.