In previous work, the IPMSRL environment (Integrated Platform Management System Reinforcement Learning environment) was developed with the aim of training defensive RL agents in a simulator representing a subset of an IPMS on a maritime vessel under a cyber-attack. This paper extends the use of IPMSRL to enhance realism including the additional dynamics of false positive alerts and alert delay. Applying curriculum learning, in the most dificult environment tested, resulted in an episode reward mean increasing from a baseline result of -2.791 to -0.569. Applying action masking, in the most dificult environment tested, resulted in an episode reward mean increasing from a baseline result of -2.791 to -0.743. Importantly, this level of performance was reached in less than 1 million timesteps, which was far more data eficient than vanilla PPO which reached a lower level of performance after 2.5 million timesteps. The training method which resulted in the highest level of performance observed in this paper was a combination of the application of curriculum learning and action masking, with a mean episode reward of 0.137. This paper also introduces a basic hardcoded defensive agent encoding a representation of cyber security best practice, which provides context to the episode reward mean ifgures reached by the RL agents. The hardcoded agent managed an episode reward mean of -1.895. This paper therefore shows that applications of curriculum learning and action masking, both independently and in tandem, present a way to overcome the complex real-world dynamics that are present in operational technology cyber security threat remediation.
In previous work, the IPMSRL environment [
RL agents will often be significantly limited if they are not exposed to the environment in which they are intended to be deployed. This also translates to when trained agents are deployed in a real scenario. Consequently, the environment needs to replicate the real scenario as closely as possible.
This paper extends the use of IPMSRL to enhance realism including the additional dynamics of false positive alerts and alert delay. Additionally, three configurations of the environment of varying degrees of dificulty are defined and tested to understand the diferent levels of performance a trained RL agent can reach.
This paper also applies curriculum learning and action masking as ways to mitigate the increased levels of dificulty, showing that using these techniques is data eficient and leads to higher mean episode reward.
Curriculum learning alone is shown to increase mean episode reward [
Finally, both curriculum learning and action masking are applied together. This training method resulted in the highest level of mean episode reward observed in this paper.
RL is a training method where an agent learns to interact with an environment to complete a task.
The environment is a Markov Decision Process (MDP) and consists of a state space, action space, reward
function, and a transition model [
IPMSRL [
This sequence is then repeated until all critical nodes are compromised which would result in a
negative reward or all infections are completely removed which would result in a positive reward. We
also implemented an early stopping criterion of 50 timesteps.
1© 2024 The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation.
2.3. PPO
Proximal Policy Optimisation (PPO) is one of the current state-of-the-art algorithms used within RL and
Multi-Agent Reinforcement Learning (MARL) [
Curriculum Learning and Action Masking are two popular guided RL methods to address data eficiency
concerns in RL [
Curriculum Learning (CL) in RL is the process of increasing the dificulty of a task by periodically
shaping aspects of the MDP throughout training [
CL has been shown to reduce learning time and improve performance of the trained agent for
applications including robotics and games [
Action Masking (AM) is a guided RL method which allows the integration of additional human knowledge
into the learning process [
Action masking limits the set of actions that can be executed per timestep based on the current
environment state. In the context of masking for discrete spaces, this is the process of reducing the
choice of actions so that undesirable or impossible actions are unavailable to the agent. Specifically, this
is achieved by setting the probabilities of selecting the undesirable actions to zero or near zero during
stochastic learning [
Previous work has shown action masking can simplify learning for the agent and result in reduced
training times in video games including StarCraft II and DOTA 2 [
The data eficiency benefits of action masking in cyber security applications with discrete action spaces were explored in this paper. Specifically, we mask invalid actions on the IPMSRL environment and show that the learning process provides a higher average return, as the agent focuses on learning only from the valid set of actions. This helped prevent the agent from wasting time exploring trajectories and taking undesirable actions that would not be applicable for deployment. In future work, the use of action masking could be further extended to restrict available actions based on safety-critical criteria.
We show below how action masking can also be applied to improve the realism of the IPMSRL simulation, including full details of how masking was applied in our experiments. For example, in the real world if the agent has sent a command to contain a node, then the user would likely have to wait for this process to complete before sending a diferent command to the given node.
AM and CL often aim to achieve the same benefits of safer learning and reduced training time. We show
below how these methods can be combined to improve performance over each method individually.
Existing work has shown how automatic action masking can be used as a type of CL to alter the action
space [
In a real scenario, Security Information and Event Management (SIEM) systems are used to give increased visibility of an OT system and flag any potential malicious activity. SIEMs are not perfect and sufer from False Positives (FP) and False Negatives (FN). The IPMSRL environment used in this paper has the additional feature of FP alerts.
Three defined environment dificulties were explored: easy, medium and hard: Table 1 shows the
values chosen for each parameter and dificulty. The easy configuration has no FP or FN alerts, no
alert delays and a 100% action success rate. The medium and hard configurations add dificulty to the
environment by amending the FP, FN, action success probabilities and the alert delay. In previous work,
it was demonstrated that as alert and action success probabilities decreased (FN increasing), a reduction
to the agent’s performance was observed [
The alerts in the IPMSRL environment are based on the MITRE ATT&CK ICS Tactics [
The alert delay parameter breaks the 12 tactics into 3 sections, displayed in Table ??. In the easy dificulty environment there is no delay present, in the medium dificulty environment there is a small delay to alerts earliest in the attack on a given node, the hard dificulty environment has a larger delay for the earliest stage tactic alerts and a small delay for tactics level 5-8. The rationale behind earlier stage tactics receiving a larger delay is that, generally, these tactics are likely to have a higher threshold which will need to be reached before malicious activity on a given node or network can be detected and reported as an alert, as the activity associated with these tactics is harder to diferentiate from user activity. Therefore, as the environment increases in dificulty, the alert delay increases towards a more realistic scenario.
It is necessary to point out that although IPMSRL has added support for more realistic and complex dynamics, it is still an abstract representation of an IPMS and an attack on this system. The diferent dificulties of environment add this realism to test the performance of diferent training approaches and algorithms, but further work needs to be completed on IPMSRL before it can be considered representative of a real-world system.
All of the experiments conducted in this paper use a PPO [
A basic hardcoded defender was designed with logic developed alongside a cyber security expert. This
hardcoded defender is not intended to be perfect, but in the absence using cyber security experts to
act as the defender remediating the threat within the IPMSRL environment, the hardcoded defender
deploys solid logic based on the NIST SP-600-61 guidance [
7. Wait.
The logic follows the fundamental idea of sequentially containing, eradicating and then recovering a node, with some modifications to prioritise recovering ofline critical nodes and remediating nodes that are ‘closer’ to critical nodes first.
The hardcoded defender was able to reach a mean episode reward over 10,000 episodes of 0.988 in an easy environment, 0.883 in a medium dificulty environment and -1.895 in a hard environment configuration. Figure 9 shows the comparison of the best performing RL defenders and the hardcoded defender.
The baseline results of a single agent acting in the IPMSRL environment with varying degrees of dificulty are shown in Figure 2. In the easy configuration, the agent reached near optimum performance after 1 million timesteps, winning every episode and achieving an extremely high episode reward mean value of 0.977. For the medium configuration, the agent reached an episode reward mean of 0.104 and in the hard configuration, the agent struggled to perform well, resulting in an episode reward mean of -2.791. These baseline results for vanilla PPO are all below that of the episode reward mean achieved by the hardcoded defender.
The results in Figure 3 show that there are clear challenges for the agent to learn an optimum policy
using a standard training process with PPO [
As discussed in the background section, CL is a training method which, in this implementation, allows the agent to initially explore simpler tasks where the agent is expected to explore more efectively, before the task is changed to a more dificult one. This enables the incremental increase in the dificulty of the environment during the agent’s training.
The tasks can be changed at pre-defined points based on either standard RLlib training metrics e.g.
mean episode reward, or custom metrics which are calculated in the IPMSRL environment e.g. win rate.
This is implemented through RLlib’s TaskSettableEnv API [
For all the CL results reported in this paper, the number of total timesteps was used to change tasks in each training sample. The number of total timesteps was chosen to be the point at which the agent, at the current task dificulty, had reached a ‘stable’ level of performance e.g. the agent’s mean episode reward had plateaued. Therefore, for diferent training curricula, tasks are changed at diferent points depending on how quickly the agent can reach a stable level of performance. During this experiment, the tasks were changed at 850k timesteps and 1.7m timesteps. Approximately a third of the total training was completed at each task dificulty, once the task’s training had converged. The light blue lines in Figure 5 show the points at which the tasks were changed, and performance subsequently dropping sharply. Figure 4 shows the curriculum used in this experiment. Figure 5 compares the results of an agent trained on the hard environment configuration (Figure 3), with an agent trained via a curriculum of easy, medium then hard dificulty configurations.
In Figure 5, it is clearly shown that the agent trained via CL can reach a significantly higher level of
performance within the training time when compared to the baseline of “vanilla” training of a PPO agent
in the hard environment configuration. The episode reward mean reached was -0.569 in comparison to
the baseline of -2.791. The CL agent also significantly outperforms the hardcoded defensive agent’s
performance in a hard environment, where it scored an episode reward mean of -1.895. This behaviour
is expected and supports the related literature’s conclusions [
The implementation of action masking encompasses two primary components: modifications within
the environment and the development of custom models that incorporate action masking. The
implementation was adapted from examples provided within RLlib [
Action masking utilises a binary array (mask) to identify permissible actions at each step. For this purpose, a specialised class is created, which, upon initialisation, reads a configuration file to ascertain the applicable mask conditions. This class features a method that is called at every environment step to evaluate which actions should be masked based on the current state of the environment. The observation space, structured as a dictionary, facilitates the mask’s transfer to the model, by including “action mask” and “observations” keys. The “action mask” is a 1D binary array indicating valid and invalid actions, while “observations” provide a conventional representation of the environment’s state.
For the custom model incorporating discrete action masking a PyTorch implementation is used, which
is compatible with RLlib and functional for DQN and Policy-Gradient style algorithms, such as PPO
[
All instances of action masking reported in this paper used the masking conditions based on input from a cyber security expert to reflect realistic cyber defence constraints and logic. These masks are deliberately simple to avoid over-engineering and unnecessary bias. The implemented masks are: • If there is no alert on an infectable node, mask the contain and eradicate action on that node. • If the infectable node is contained, mask the contain action on that node. • If the infectable node is not contained, mask the eradicate and recover action on that node. • If an action is already in progress on an infectable node, mask all actions on that node. • If a critical node is online, mask the recover action on that node.
Figure 6 shows the baseline results of the easy, medium and hard environment configurations when action masking is applied, compared to the baseline results without action masking. In all instances, both a dramatic improvement in data eficiency and overall performance compared to vanilla PPO can be seen. In the easy configuration, for all seeds, the agent reached an optimum level of performance in less than 100k timesteps, winning every episode, resulting in an episode reward mean of 0.977. In the medium dificulty configuration, the agent reached a good level of performance, an episode reward mean of 0.816.
In the hard environment configuration with an action mask, the agent reached a higher level of performance than solely training on the hard environment, as shown in Figure 7, with an episode reward mean of -0.743 in comparison to the baseline hard environment episode reward mean of -2.791. The agent trained with an action mask achieved a slightly lower mean episode reward mean than CL, but was able to reach that episode reward mean at a sharper rate of learning. The AM agent, similarly, to the CL agent, significantly outperformed the hardcoded agent in the hard environment.
Following the conclusions of the previous sections in this paper AM and CL were applied together with the subsequent curriculum shown in Figure 8.
The curriculum outlined in Figure 8 shows that the tasks are switched much earlier than the curriculum for the experiment without action masking presented in Figure 4. This is because, as shown in Figure 6, the learning of policies trained with action masking plateaued far sooner than policies trained without action masking. The curriculum is consequently adapted to match the attributes of agents trained with action masking, changing tasks at 250k and 750k timesteps.
In Figure 9 it can be observed that when the techniques of CL and AM are combined, for all seeds, an agent trained in these conditions reached a higher mean episode reward of 0.137, in fewer timesteps, than agents trained solely with CL, AM or without either. The combination of CL and AM was therefore shown to produce the highest level of performance in comparison to the other training techniques tested and the hardcoded defender.
This paper demonstrates that the application of action masking and curriculum learning individually improved the overall performance of training a defensive agent to remediate attacks in more complex IPMSRL environment configurations. This includes real work dynamics such as false positive and negative alerts and the delay that is inherent in OT systems. The benefits of applying these techniques were even more pronounced when they were applied together.
Curriculum learning alone was shown to increase episode reward mean from -2.791 to -0.569. Action masking alone is shown to similarly improve mean episode reward. In this paper an improvement to -0.743 was seen.
Finally, both curriculum learning and action masking are applied together. This training method resulted in the highest level of mean episode reward observed in this paper, with a mean episode reward of 0.137.
As the complexity of the environment increased, the hardcoded defender struggled to maintain a strong level of performance. The hardcoded defender was able to outperform vanilla PPO, but the application of AM or CL training techniques enabled the RL defensive agent to achieve a significantly higher episode reward mean than the hardcoded defender in the hard dificulty environment.
A potential reason that the hardcoded defender struggled in the hard environment, in comparison to the best performing RL agents, is the uncertainty that a high proportion of FP alerts provides. The hardcoded agent struggles to prioritise the remediation of true alerts as there is no mechanism in its logic to establish whether an alert is a FP or not. It will have to randomly choose between which alert to remediate if there are multiple alerts active. The only prioritisation that is baked into the logic of the hardcoded defender is to focus on nodes adjacent to critical nodes. The RL agents on the other hand may have developed policies that allow them to more eficiently decide which alerts to prioritise. This behaviour is very dificult and time consuming to encode into a hardcoded defender’s logic, further displaying the benefits of using an RL based approach for autonomous cyber security.
An important note about the use of curriculum learning is the significance of defining appropriate task changing criteria. This paper implemented a simple curriculum, changing tasks at a specified number of total timesteps. There is further scope to optimise this process and potentially see higher levels of performance.
Additionally, there is a trade-of present when using action masking; the masking conditions used during training need to be present when querying the trained policy. This is a drawback in the sense that it adds a dependency to the agent’s deployment, and additional bias is added through the setting of masking conditions. But the benefit of constraining certain actions which don’t meet the requirements set out when developing the masking conditions is a tangible one. The use of action masking in this way therefore benefits from gains in data eficiency, overall performance, and an ability to restrict actions to meet the user-defined requirements of a given system. In future work the use of action masking could begin to consider the safety-critical nature of OT systems. Action masking could further be used to help to build trust in autonomous agents that aim to be applied to real systems.
Research funded by Frazer-Nash Consultancy Ltd. on behalf of the Defence Science and Technology Laboratory (Dstl) which is an executive agency of the UK Ministry of Defence providing world class expertise and delivering cutting-edge science and technology for the benefit of the nation and allies. The research supports the Autonomous Resilient Cyber Defence (ARCD) project within the Dstl Cyber Defence Enhancement programme.
The authors would also like to thank Lisa Gralewski, Marco Casassa Mont, David Foster, Clare Jubb, Laura Caddy, Tasha Hughes and Jake Rigby for their wider contribution to the project and paper.
Hyperparameters Values