This paper demonstrates the potential for autonomous cyber defence to be applied on industrial control systems and provides a baseline environment to further explore Multi-Agent Reinforcement Learning's (MARL) application to this problem domain. It introduces a simulation environment, IPMSRL, of a generic Integrated Platform Management System (IPMS) and explores the use of MARL for autonomous cyber defence decision-making on generic maritime based IPMS Operational Technology (OT).
CEUR ceur-ws.org
Operational Technology (OT) cyber defensive actions are less mature than they are for Enterprise
IT. This is due to the relative ‘brittle’ nature of OT infrastructure originating from the use of
legacy systems, design-time engineering assumptions, and lack of full-scale modern security
controls. Traditional IT controls are rarely deployed on OT infrastructure, and where they
are, some threats aren’t fully addressed. Additionally, there may be a lack of trained cyber
CEUR
Workshop
Proceedings
personnel available in deployed operational OT environments e.g., at-sea vessels, hence the
opportunity to develop autonomous defensive systems. Reinforcement learning (RL) [
RL has seen key recent developments in various domains such as game environments e.g.,
Go [
We applied Multi-agent Reinforcement Learning (MARL) to the cyber defence of ICSs because of the collaborative nature of the problem. Agents working together distributed across a network taking coordinated remedial actions is likely to be more successful than a single, or multiple uncoordinated, agent(s) working independently.
This paper focused on a generic IPMS, a form of Industrial Control System used onboard ships and submarines.
An IPMS provides the capability for remote monitoring, control and management of the ship’s machinery systems and damage control from key components. This real-time monitoring and control facilitates continuous situational awareness of the machinery state and damage condition of the ship.
IPMS controls and monitors many ship systems across Propulsion, Power, Steering, Stability, Auxiliary and Ancillary systems as well as those providing Damage Control and Fire Fighting (DCFF). To achieve this, IPMS utilises a distributed control system architecture that facilitates interfaces with sensors, equipment, plants, software-based control systems and network-based data.
An IPMS produces alerts in event of failures, anomalies, or issues. These alerts are not necessarily cyber security alerts; however, they are likely to be instrumental in the context of cyber-attack detection and response. These alerts were assumed to be fed into additional Cyber Threat Detection and Security Information and Event Management (SIEM) tools, leveraging IPMS inputs to detect and generate relevant attack alerts. The IPMS architecture is based upon traditional Industrial Control System (ICS) design, inclusive of Remote Terminal Units (RTUs), Programmable Logic Controllers (PLCs) and multifunction control consoles providing Human Machine Interfaces (HMIs) combined by a dual redundant network backbone. Specifically, HMI refers to IPMS consoles as well as panels on the equipment. The architecture work has been deliberately abstracted and generalised using open-source information.
In this paper we introduce IPMSRL, a highly configurable network based multi-agent RL environment. IPMSRL simulates an abstracted generic maritime IPMS where defensive agents attempt to restore an infected network. Meanwhile an attacker attempts to propagate through the network and disrupt IPMS and controlled systems to negatively impact operations.
The IPMSRL network comprises two types of node: critical nodes and non-critical infectable nodes. Critical nodes represent components of core-controlled systems critical to efective operation of the vessel. Two controlled systems are modelled, Chilled Water Plants (CWPs) and the Propulsion System. This selection represents a diverse subset of available controlled systems. Infectable nodes represent HMIs, RTUs, Local Operating Panels (LOPs), PLCs, and network switches.
The properties of the links between nodes, shown in Figure 1, difer depending on their type. In Figure 1, the star representation of the network is for visualisation purposes only. Any HMI/RTU/LOP node which is connected to a network switch is adjacent to any other node directly connected to a network switch, i.e., they are interconnected. The network holds redundancy in its structure through its dual ring network backbone.
MITRE ATT&CK1® ICS [
Fine-grained attack steps, based on MITRE ATT&CK® ICS, were implemented to enable more realism, context and complexity to the cyber defensive remedial actions. At diferent MITRE ATT&CK® ICS stages the attacker will have diferent capabilities e.g., lateral movement by leveraging an HMI’s network card and connections. Diferent defensive remedial actions are required depending on the MITRE ATT&CK® ICS stage that the attacker has exploited.
The twelve attack stages, represented in IPMSRL, correspond to the associated MITRE ATT&CK® ICS Tactics are shown in Figure 2:
The goal of the attacker is to compromise the vessel’s operational capacity, for example, by
disrupting IPMS systems and controlled systems (CWPs, Propulsion System) which negatively
impacts the vessel. An attacker propagates through the network by initially infecting a
noncritical infectable node, increasing the node’s infection level using an abstraction of the MITRE
ATT&CK® ICS [
Attackers can be created on a sliding scale from fully targeted to fully viral. Fully targeted attackers move directly towards critical nodes displaying ‘knowledge’ of the network. Fully viral attackers move randomly to any adjacent node. Partially viral attackers move towards critical nodes but may also move randomly with a given configured probability.
Attackers also have other configurable attributes such as the probability an infection will progress through the MITRE ATT&CK® ICS stages or the probability of a successful lateral move.
Each of the infectable nodes has an alert system which can be triggered when an infection is present or when an attacker initially tries to infect a non-infected node. The probability that an alert is successfully parsed to the defender(s) allows the user to configure the scale of asymmetric information within the environment. In the context of this paper, an assumption was 1© 2023 The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation. made that IPMS-generated alerts were fed into a SIEM for further processing and cyber threat detection. The alert success probability, therefore, directly impacts the partial observability of the defender’s observation space.
It is worth noting that the alerts are static in nature. This means that if an alert is set of at a
certain timestep, the defender will receive information about the progress of the node infection
at a given timestep. No further information about the progression of the infection will be given
to the defender unless another alert is set of or the defender takes a remedial action on a node.
3.4. NIST SP-800-61
NIST SP-800-61 [
Each defensive agent can take four types of action: Contain, Eradicate, Recover and Wait. These are aligned with the real-world cyber incident response process indicated in NIST SP-800-61 discussed above.
Containing makes a given node un-operational but prevents the infection from propagating from that node. Eradicating cleans a node of infection. Recovering returns the node to an operational state. For critical nodes, only the recover and wait actions are available whereas, for all other nodes, all actions are available. At each timestep, each defender in succession takes an action. Each of these actions have a configurable delay where a timestep lag is applied depending on the infection state of the node and type of action. This delay tries to reflect the real-world diferences between taking simple actions against early infections e.g., modifying credentials, and taking more extreme actions to prevent unrecoverable damage e.g., quarantining systems. Additional dialogue on valuing the costs of remedial actions is considered in later discussion about the reward function.
IPMSRL supports partial observability in which each defensive agent has its own view of the network state. In addition to the alerts, defensive agents can investigate a node’s true infection information by interacting with that node. This knowledge is not shared amongst other defensive agents.
IPMSRL features a highly customisable reward function for both global and intrinsic rewards.
Global rewards are awarded to the agents at the end of the episode, whereas intrinsic rewards
are awarded within an episode2. Global rewards have been subdivided into mission objective
and state rewards. The implemented reward function aims to tackle the problem of sparse
global rewards, where agents struggle to train since they only receive a small number of reward
signals, often at the end of an episode. This is a known problem within RL, and specifically
within applied control problems such as robotics [
• Draw (+0) – = and neither the win nor loss criteria has been met for t steps. For our experiments, = 50 .
The state reward reflects the impact to non-critical and critical nodes during the episode and is graded as low, medium or high. Critical nodes, such as PLCs, RTUs and PCSs that directly control machinery or switches, are more strongly penalised than other nodes, such as HMIs. 2The global and intrinsic rewards are divided and shared equally between each agent.
The weighting of penalties was informed by the failure efects and impact analysis of a generic IPMS. State reward is produced by a 50/50 split of ‘state generic’ and ‘state specific’ rewards:
State Generic – this refers to penalties that are provided when nodes within the environment meet certain conditions e.g., a node is contained but not infected or a node is uninfected and contained without being recovered.
State Specific – this is defined, in levels of severity, as the number of diferent types of nodes which had been infected in an episode. For example, if two HMIs were infected then the smallest negative reward value (defined in the config) is applied.
IPMSRL has the ability for agents to receive intrinsic rewards called action score rewards. The defender(s) receive a negative reward depending on the action chosen and the status of the node that is being interacted with (graded as low, medium or high). Nodes with a more severe infection status receive a higher penalty. The agents are therefore incentivised to deal with infections early when they are first alerted. The weighting of the reward function components are user configurable and will be discussed in further detail within the experimental results section.
In this paper, two MARL algorithms, IPPO [
An Initial hyperparameter tuning stage to create a baseline for the environment parameter experiments was completed. A brief grid-search over commonly successful hyperparameters was 3Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Q-MIX were also explored but the implementations within Ray and RLlib were found to be unstable during training on IPMSRL. 4This is the same as the implementation within MARLlib [15], using the global state within training. used for eleven PPO specific and general machine learning hyperparameters. In Figure 4, a clear diference between the training performance between the tuned and default hyperparameters can be seen. The hyperparameters which were most influential in the improved performance were the Learning Rate, Gamma, Lambda and the Clip Parameter. When MAPPO was trained using the default hyperparameters, it converged to an optimal policy more slowly than the tuned model, but it was still able to reach an optimal policy. MAPPO was also used during the Experimental Results section. Throughout the experiments, it was found that in general, PPO based algorithms were still able to improve their performances during training with most configurations of hyperparameters.
MAPPO outperformed IPPO, converging to the optimal policy quicker. This statement is based
on the higher win rate, higher reward total and the ability to win an episode in fewer episode
steps, suggesting a more eficient strategy has developed. The point at which the two algorithms
diverge in Figure 5 underpins some of the diferences in performance. At 200K timesteps IPPO
and MAPPO have similar performance and are drawing most episodes. After this point, both
IPPO and MAPPO start to develop a winning policy but with MAPPO optimising its policy at a
faster rate. In our experimentation, MAPPO clearly gains an advantage using the same critic
network. The centralised critic allows for the agents to find a better policy where agents are
collaborative faster than without the shared critic network. This is the converse to what has
been found elsewhere in the literature, where the addition of a shared value function reduced
the performance significantly [
MAPPO’s confidence interval (90%) bands were much narrower than IPPO. The narrower area of the confidence interval suggests that the individual trials of MAPPO algorithm were more stable than IPPO, implying the training is more consistent. This experiment demonstrates that the centralised critic can help improve the policy updates at each training epoch.
A configurable attacker parameter is the alert success probability: this is the probability of an alert being set of on any infected node after an infection on a node progresses successfully laterally to another node. environment is clear. The more accurate the information they receive, the better they can perform. Without an alert success probability of 1, within the training time set, none of the parameters were able to produce an optimal strategy. However, an agent with an alert probability of 0.75 or 0.9 was still able to win in over 97.5% or 99.5% of episodes, respectively. These results suggest that the agents can learn efective strategies in partially observable environments, but that the SIEMs alert detection and processing is important to the success of autonomous cyber defence.
In Figure 7, an investigation into the diference in training performance between state-based rewards and balanced rewards was conducted. State based reward refers to a reward function solely made up of the state rewards discussed in the reward function section. Whereas balanced reward also includes intrinsic rewards, given to the agents during an episode based on their actions, and an overall score based on the episode outcome. Using solely state rewards, convergence towards a winning strategy was faster than for balanced rewards, but performed less optimally overall than balanced rewards. Figure 7 shows the two parameter sets plotted with 90% confidence intervals, validating their significant diferences in behaviour.
This paper introduced IPMSRL, a highly configurable network based multi-agent RL environment and demonstrated the capabilities of MARL agents to successfully recover an abstract IPMS to an operational state following a cyber-attack. Findings demonstrate that hyperparameter tuning, reward shaping and the quality of alerts are all important aspects in developing performant agents.
A key issue with using a simulator to train an RL agent is the sim-to-real-gap. This describes the diferences present between real systems and the simulated environments that are used to replicate them and the problems that even small diferences can cause. The issues are compounded further when environments are intentionally abstracted away from the real system for reasons of dimensionality, security or complexity. RL simulators such as IPMSRL are important for developing the core concepts needed to build confidence and understanding in the techniques being employed, such as MARL. But more realistic simulators or emulators will be required to facilitate movement towards applying these concepts on real systems, which should be the eventual goal.
Another avenue of future research is the generalisability of the agents. Agents will need to be able to adapt to diferent types of attack, scenario (where components may be valued diferently) and network topology. Without demonstrating this flexibility, agents are unlikely to be stable enough to be trusted in real world control systems.
Research funded by Frazer-Nash Consultancy Ltd. on behalf of the Defence Science and Technology Laboratory (Dstl) which is an executive agency of the UK Ministry of Defence providing world class expertise and delivering cutting-edge science and technology for the benefit of the nation and allies. The research supports the Autonomous Resilient Cyber Defence (ARCD) project within the Dstl Cyber Defence Enhancement programme.
The authors would also like to thank Clare Jubb, Andy Pollard, Christos Giachritsis, Jake Rigby, Ben Golding, Julie Kimbrey and Brian Bassil for their wider contribution to the project and paper. Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?, 2020. URL: http://arxiv.org/abs/2011.09533, arXiv:2011.09533 [cs]. [14] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, Y. Wu, The Surprising Efectiveness of PPO in Cooperative, Multi-Agent Games, 2022. URL: http://arxiv.org/abs/2103.01955, arXiv:2103.01955 [cs]. [15] S. Hu, Y. Zhong, M. Gao, W. Wang, H. Dong, Z. Li, X. Liang, X. Chang, Y. Yang, MARLlib: A Scalable Multi-agent Reinforcement Learning Library, 2023. URL: http://arxiv.org/abs/ 2210.13708, arXiv:2210.13708 [cs]. [16] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, I. Stoica, Tune: A research platform for distributed model selection and training, arXiv preprint arXiv:1807.05118 (2018).