Distributed control is a reality of today's industrial automation and systems. Parts of a system are on-site, and other elements are on the edge of the cloud. The overall system-functioning relies on the reliable operation of local and remote components. However, all system parts can be attacked. Typically, local entities of a cyber-physical system, such as robot arms or conveyor belts, get afected by cyber attacks. However, attacking the control and monitoring channels between a plant and its remote controller is attractive, too. There is a diversity of attacks, such as manipulating a plant's input signals, controller logic, and output signals. To detect and mitigate the impact of such various attacks and to make a plant more resilient, we introduce a self-learning controller proxy in the plant's communication channel to the controller. It acts as a local trust anchor to the commands received from a remote controller. It does black box self-learning of the controller algorithms and audits its operations. Once an attack is detected, the plant pivots into self-piloting mode. We investigate design alternatives for the controller proxy. We evaluate how complex the control algorithms can be to enable self-piloting resilience.
A Cyber-Physical System (CPS), such as a critical infrastructure, consists of distributed physical elements, including local and of-site control algorithms that work together to accomplish a task. All together, the physical elements form the plant. The algorithms make up the controller. Controller and plant communicate through a network, constituting a Networked-Control System (NCS). It is a flexible environment but vulnerable to attacks: attacks on one component can afect the entire system. Hence, the importance of securing a NCS.
The security of a NCS encompasses several aspects, including protection against attacks, recognition of attacks and incident response. Protection is realized by leveraging cryptography techniques and security protocols. Recognition of attacks leverages tools such as intrusion detection techniques. In this paper, we explore an idea for preparing responses to attacks. In particular, when the severity of attacks is such, it becomes safer for a plant component to disconnect from the network and operate autonomously, at least for a short while.
We propose a method to automatically configure a local model of a remote controller by observing its actions. In case a network disconnection is required, the resulting local controller proxy can take over and assure the operation for some time. It makes the local infrastructure more resilient to cyber-attacks. Starting from a mathematical model, the paper develops an approach to learning and consequently predicts a remote controller’s behavior. The resulting algorithm is evaluated with an emulated industrial control process.
The related work is reviewed in Section 2. Our system model is described in Section 3. In Section 4, we discuss learning by imitating. Our approach is evaluated in Section 5. We conclude with Section 6.
A CPS is a combination of hardware and software resources, collectively called a plant or system.
The term NCS refers to a CPS monitored and controlled remotely through a communication
network. Autonomous transport systems and energy distribution systems are NCS examples. A
CPS is vulnerable to several availability, covert [
Protection involves using cryptographic methods such as digital signatures, encryption, and
key establishment. Despite protection, impactful attacks will likely be perpetrated. Hence,
detection methods and response plans are required. Several methods have been proposed
to detect attacks on a CPS, such as challenge-response authentication [
Our work is related to model predictive control and learning. In predictive control, at time ,
the controller sends an input row vector ⃗ of length k+1, the time horizon. The first element of
⃗ is the input to apply at time . Predicting a response from the plant (using a model of it), the
vector ⃗ also contains the expected inputs for times + 1 to + . When the plant receives an
input vector ⃗, it applies the first element in normal mode. The remaining elements are stored.
When a situation occurs, the plant stops accepting new inputs. It keeps running, applying the
inputs predicted by the controller. The approach has been used by Quevedo et al. and Franzè et
al. to make a system resilient to packet loss [
A CPS is modeled by the following two discrete time equations: +1
= (, ) = () and are the current state and actuator inputs, at time , with ∈ R and ∈ R. +1 is the successor state determined by the evaluation of the state-input function (, ), at time + 1, with +1 ∈ R. is the sensor outputs in the current state defined by the evaluation of output function (), with ∈ R. The variables , , and denote positive integers.
As an example, let us consider a system that consists of a single cylindrical tank. The tank
has one inflow and one outflow of liquid. The dynamics of liquid level in the tank are captured
by the following diferential equation [
Eq. (3) models the relationship between instantaneous changes in liquid level and the diference between the inlet flow rate and outlet flow rate. As a function of time (second), the level of the liquid in the tank is ℎ() (cm). Variable represents a cross-sectional area of the tank (cm2). The term () represents the inlet flow rate (cm 3/second). The parameter denotes the outlet valve coeficient. The outlet flow rate (cm 3/second) is proportional to the product times the square root of the pressure represented by the term ℎ (). Note that because of the square root term, the system is nonlinear.
Mapping Eq. (3) into the model of Eq. (1) , we get: +1
= (, ) = + = () = − √ where represents the input flow rate at time . In the sequel, this dynamics is used to build a NCS case study.
We assume that remote monitoring control of the CPS is required. Hence, it is a NCS. The architecture of a NCS is pictured in Figure 1 (a). There is a System consisting of resources that can be controlled and monitored. There is a Controller that posts inputs and gets outputs from the System. The inputs are commands to actuators of the System. The outputs are readings from sensors attached to the System. Inputs and outputs travel over a network. This architecture is relevant to situations requiring remote control and monitoring. The attack model is pictured in Figure 1 (b). Somewhere in the network, an adversary stands between the Controller and System. The adversary can modify inputs and outputs. In the sequel, we assume that data modification attacks can be detected by both the Controller and System, using, for instance, a digital signature mechanism. (1) (2) (3) (4) (5)
(b) Attack model.
We define resilience as a system’s ability to maintain operation while an attack is being perpetrated. In Figure 1 (c), we propose a system architecture with resilience by design. The remote controller controls and monitors the system during regular operation. The system can detect attacks from cyberspace and disconnect itself from the network when attacks occur. When the perpetration of an attack is detected, the System disconnects itself from the network. The System embeds a controller proxy. While disconnected from the network, control of the CPS is handed over to the controller proxy. The controller proxy may act on the physical system with its actuators and observe it with its sensors. Hence, the controller proxy may have its own state-input function (, ) and output function ().
We develop a controller proxy learning approach for the architecture of Figure 1 (c). Let us ifrst consider radio-controlled model airplanes. They have a pre-programmed solution. They memorize their launching location. When a model airplane goes out of the wireless range of its controller, the aircraft detects the signal loss. An auto return to launch site function kicks in. The plane returns over the takeof point, circles around it, and lands safely. The return to launch site solution has also been adopted by modern quadcopters. This solution is interesting and specific to a context. It can hardly be generalized to other types of situations. We aim at general solutions.
We follow the more generic idea of imitation learning [
We propose a flexible solution where a controller proxy observes, imitates, and acquires the behavior of a remote controller, i. e., the actions it performs in each state. In this context, imitation learning means that the controller proxy is an agent that infers the controller’s behavior. The remote controller is a teacher. The controller proxy is a learner. The teacher does not need to be aware of the learner. It performs its task normally. The learner is an observer. The teacher is part of its environment. When an attack is detected, the controller proxy agent acts on the system instead of the remote controller. For detection methods, see Section 2. In the sequel, we focus on the controller proxy agent training.
The agent can only observe the communications between the controller and the system. There are two types of communication. The first type of communication is from the controller to the system. This communication contains commands to the system. The system executes the commands and responds with the resulting state. As a function of the new state, the controller sends a new set of commands in the second communication type. To faithfully imitate the controller, the controller proxy needs to reliably predict the new set of commands based on the reported system state. Therefore, it must learn a policy that reflects the controller’s decision-making.
Our reinforcement learning architecture reflects this setting. There are two types of scenarios. The first type of scenario is stateless. The controller consistently makes the same decision given the same input. The second type of scenario is stateful. In this case, in addition to the system’s current situation, the state also needs to capture the trends of the ongoing activity. For example, a water tank may have a changing target level. A controller’s decisions for a water tank collecting water difer from the commands for a tank leaking water. Because they are more general, we focus on stateful physical systems.
The input to the agent is a series of observations of the communication flow on the channel between the controller and the system. Let a command from the controller to the system have parameters. Moreover, let a system state be a vector of parameters. With denoting the number of interactions between the controller and system, i. e., one command and a corresponding state vector, the input to the agent has length · ( + ) parameters. The agent is limited to the actions that the controller takes. Since the agent should imitate the controller, the commands it receives have the same dimension as the commands produced by the controller. Furthermore, the agent’s output is also command vectors of parameters, i. e., its choice of actions according to the policy acquired with reinforcement learning. The size of the reinforcement learning action space grows exponentially with the number of commands available to the controller. The growth depends on two factors. The first is the number of possibilities for each action. The second factor is the number of actions included in a command vector. The combination of
Upper tank Pump
Lower tank possible commands, i. e., actions, grows with the possible combinations that the controller can send. When receiving an input, the agent predicts the chosen action of the controller. When it correctly predicts the new parameters, it receives a positive reward. Otherwise, it receives a negative reward, i. e., a penalty.
We use a neural network for the core of the agent. Within these parameters, the neural network has · ( + ) input neurons and output neurons. In addition, the neural network can have one or more hidden layers of varying sizes. Their number depends on the scenario. The agent should accurately imitate the controller’s logic. This implies that the agent should not explore new action paths to optimize its policy. The agent concentrates on exploitation.
The training of the agent results in a policy that reflects the controller’s logic. If needed, it makes it capable of replacing the controller in case of an attack and managing the system according to the observed behavior. In contrast to predictive control with a finite time horizon, the controller proxy can operate while the system is disconnected for an arbitrarily long period.
In this section, we demonstrate the controller proxy concept. We assess the feasibility of our method and its performance versus the number of observations made by the controller proxy. To do so, we choose a concise yet realistic setting: a two-water tank control problem. The plant comprises two tanks while the controller maintains target water levels. The controller is remote to the plant in the cloud. Collocated with the plant, we place a self-learning controller proxy. The quality of service is evaluated with variable conditions.
The process works as follows. The controller proxy learns by passively observing the communication between entities inside the plant and one or multiple external controllers. After observing the state of the monitored plant, the internal controller decides and compares its decision to the command received from the external controller in the next step. When the external controller gets disconnected, the internal controller proxy takes over and mimics its behavior to keep the plant in an optimal state.
For our evaluation, we consider the following scenario. The plant consists of two water tanks, one being on top of the other. Two pipes connect the water tanks. One pipe has an in-line water pump that can be turned on or of. When turned on, the pump moves water from the lower tank to the upper one. While the pump moves the water to the upper tank, the water level in the lower tank decreases. The water from the upper tank flows through the second pipe from the upper tank to the lower tank. When the pump is of, the water level in the upper tank decreases and the level in the lower tank increases. The pump turns on when the upper tank is emptied to a certain threshold. It remains on until the upper tank reaches the maximum threshold and is turned of. The pump is reactivated when enough water from the upper tank has flown to the lower tank. Figure 2 shows the setup.
Figure 3 shows this deterministic cyclic behavior to be learned by observation and imitation. The horizontal axis corresponds to the time in seconds. The vertical axis represents the water level in tanks in cm. The green curve denotes the water level of the lower tank. The orange curve corresponds to the level of the upper tank. The blue line represents the state of the pump; the low level is of, high status is on. For this scenario, the sampling period is five seconds.
This scenario is implemented and simulated using the Virtual State Layer (VSL)
middleware [
We train a Deep Q-learning (DQN) agent for this task on the observing controller proxy. The input to the DQN is the current level of each water tank and the status of the pump. The action space switching the pump on or of in the next step. The DQN outputs a prediction for the activation of the pump in the next step. It also observes the command of the external controller that it predicted and uses it as label to evaluate the prediction. We implement the DQN in a separate Python program using PyTorch. The agent is connected via interfaces to the VSL middleware such that it can observe the activities in the simulation.
There are various parameters that can afect the result of the training. For our evaluation, we focus on the number of epochs for training, and the number of observations at the proxy controller. Both afect the amount of data that the proxy controller has for learning.
In a first experiment, we evaluate the quality of proposed actions by the local controller proxy depending on the number of trained epochs and observations. We assess the performance by comparing DQN predictions against the actual commands issued by the external controller.
We train multiple DQNs while varying amounts of data and epochs. Each DQN has random weights at initialization. Then, we train each DQN for 1, 2, . . . , 14 epochs. We repeat this experiment with diferent amounts of training data. We use sequential data starting from the beginning of the simulation. Depending on the amount of provided data, a DQN is trained with 0.9 0.8 led0.7 o m f o cya0.6 r u c c A 0.5 0.4 800 600 400 200 0 ) m c ( l e v e l r e t a W 0 1000
2000 3000 Observations used for training 4000 a partial observation of a cycle, a complete, or multiple cycles. The smallest amount of data used for training is the first 100 measurements. For the number of epochs, the DQNs are trained with 100, 200, 300, . . . , 4300 observations.
Figure 4 shows the accuracy of DQNs as a function of the number of trained epochs averaged over the diferent amounts of training. The accuracy is defined as correctly predicted labels over all samples. The accuracy rises until epoch seven. DQNs trained for eight to 10 epochs have decreasing accuracy. DQNs trained for 12 epochs have the highest accuracy. Considering the trade-of between training time and accuracy, we use seven epochs for further evaluation. The accuracy gain at 12 epochs is only around 5%, while the training time rises by about 170%.
Using the fixed number of seven epochs, the following experiment evaluates the impact of the number of observations on a model’s accuracy. We, therefore, trained multiple DQNs while varying the amount of sequential training data. Figure 5 shows that the performance of trained models correlates with the quantity of training data. The dashed lines show the observations used to train the model. Models that are only trained on observations where the pump is turned of perform poorly. The performance increases significantly after observing one cycle in which the pump was turned of and turned on. The maximum accuracy is achieved at 3500 observations. The results are averaged over ten iterations of the experiment.
We implemented in VSL the internal controller proxy to verify the training accuracy. We used a learned model on live data. Based on the results of the former evaluation, the controller proxy implements a model trained for up to seven epochs, using 3500 observations. The results
Pump activation Upper tank
Lower tank 0 500 1000
Time (s) 1500 2000 are shown in Figure 6. The red lines mark the upper and lower limits implemented in the original controller. Compared to Figure 3, it shows that the internal controller proxy learned the behavior of the external controller. The internal controller keeps the upper tank level within the limits of the external controller.
The previous evaluation shows the feasibility of the approach. Our local proxy controller became a self-learned digital twin of the remote controller. It showed that the approach works in our limited scenario. Our evaluation setting fulfills our assumptions of a local CPS that is controlled from the outside. Therefore, It can be expected that the approach is also fitting for bigger settings. We plan to evaluate this in the future.
This paper showed how a remote controller could be learned locally, resulting in a digital twin of the controller. It showed how the digital twin acts as a trust anchor, enabling anomaly detection and mitigation from attacks by taking over control in case of an attack.
This work shows the feasibility of the idea by describing the approach and evaluating it at a representative scenario. In the future, we plan to focus on more complex systems to evaluate the performance of the approach.
This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). This research took part in the context of the industrial chair Cybersecurity for Critical Networked Infrastructures (CyberCNI.fr) with the support of the FEDER development fund of the Brittany region. This work was also supported by the German Federal Ministry of Education and Research under funding number 16KIS1221 (SKINET),