<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Process Calculus Approach to Correctness Enforcement of PLCs?</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Ruggero Lanotte</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Massimo Merro</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Andrei Munteanu</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Università degli Studi dell'Insubria</institution>
          ,
          <addr-line>Como</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Università degli Studi di Verona</institution>
          ,
          <addr-line>Verona</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>We define a simple process calculus, based on Hennessy and Regan's Timed Process Language, for specifying networks of communicating programmable logic controllers (PLCs) enriched with monitors enforcing specifications compliance. We define a synthesis algorithm that given an uncorrupted PLC returns a monitor that enforces the correctness of the PLC, even when injected with malware that may forge/drop actuator commands and inter-controller communications. Then, we strengthen the capabilities of our monitors by allowing the insertion of actions to mitigate malware activities. This gives us deadlock-freedom monitoring : malware may not drag monitored controllers into deadlock states.</p>
      </abstract>
      <kwd-group>
        <kwd>Process calculus</kwd>
        <kwd>PLC correctness</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>Industrial Control System (ICSs) are distributed systems controlling physical
processes via programmable logic controllers (PLCs) connected to sensors and
actuators. PLCs have an ad-hoc architecture to execute simple processes known
as scan cycles. Each scan cycle consists of three phases: (i) reading of the
sensor measurements of the physical process; (ii) derivation of the commands to
guide the evolution of the physical process; (iii) transmission of the calculated
commands to the actuator devices.</p>
      <p>
        Published scan data show how thousands of PLCs are directly accessible from
the Internet [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ]. When this is not the case, PLCs are often connected to each
other in field communications networks, opening the way to the spreading of
worms such as the PLC-Blaster worm [
        <xref ref-type="bibr" rid="ref29">29</xref>
        ] or the PLC PIN Control attack [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ].
      </p>
      <p>
        As a consequence, extra trusted hardware components have been proposed to
enhance the security of ICS architectures [
        <xref ref-type="bibr" rid="ref24 ref25">24,25</xref>
        ]. In this respect, McLaughlin [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ]
proposed to add a policy-based enforcement mechanism to mediate the actuator
commands transmitted by the PLC to the physical plant, whereas Mohan et
al. [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ] introduced an architecture in which every PLC runs under the scrutiny
of a monitor which looks for deviations with respect to safe behaviours ; if the
? Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
behaviour of the PLC is not as expected then the control passes to a safety
controller which maintains the plant within the required safety margins.
      </p>
      <p>Both architectures above have been validated by means of simulation-based
techniques. However, as far as we know, formal methodologies have not been
used yet to model and formally verify security-oriented architectures for ICSs.</p>
      <p>
        The goal of the paper is to verify the effectiveness of a process calculus
approach to formalise runtime enforcement of specification compliance in
networks of PLCs injected with colluding malware that may forge/drop both
actuator commands and inter-controller communications3. Process calculi represent
a successful and widespread formal approach in concurrency theory relying on
a variety of behavioural equivalences (e.g., trace equivalence and bisimilarity)
for studying complex systems, such as distributed migrating systems [
        <xref ref-type="bibr" rid="ref10 ref6">6,10</xref>
        ], IoT
systems [
        <xref ref-type="bibr" rid="ref13 ref15">13,15</xref>
        ] and cyber-physical systems [
        <xref ref-type="bibr" rid="ref14 ref19">14,19</xref>
        ], and used in many security
fields, including verification of security protocols [
        <xref ref-type="bibr" rid="ref1 ref21">1,21</xref>
        ] and security analysis of
cyber-physical attacks [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. On the other hand, runtime enforcement [
        <xref ref-type="bibr" rid="ref20 ref28 ref8">28,20,8</xref>
        ]
is a powerful verification/validation technique aiming at correcting
possiblyincorrect executions of a system-under-scrutiny (SuS) via a kind of monitor that
acts as a proxy between the SuS and its environment.
      </p>
      <p>Thus, we propose to synthesise a proxy from an uncorrupted PLC, to form a
monitored PLC ensuring: 1. observation-based monitoring, i.e., the proxy should
only look at the observables of the PLC, and not at its internal execution;
2. transparency, i.e., the semantics of the monitored PLC must not differ from
the semantics of the genuine (i.e., uncorrupted) PLC; 3. sound execution of the
monitored PLC, to prevent incorrect executions; 4. deadlock-freedom, i.e., an
injected malware may not drag a monitored PLC into a deadlock state.</p>
      <p>
        Obviously, if the PLC is compromised then its correct execution can only be
enforced with the help of an extra component, a secured proxy, as advocated by
McLaughlin [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ] and Mohan et al. [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ]. This means that any implementation of
our proposed proxy should be bug-free to deal with possible infiltrations of
malware. This may seem like we just moved the problem over to securing the proxy.
However, this is not the case because the proxy only needs to enforce correctness,
while the PLC controls its physical process relying on malware-prone
communications via the Internet or the USB ports. Of course, by no means runtime
reconfigurations of the secure proxy should be allowed.
      </p>
      <p>
        Contribution. We define a simple timed process calculus, based on Hennessy
and Regan’s TPL [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], for specifying networks of communicating monitored
controllers, possibly injected with colluding malware that may forge/drop both
actuator commands and inter-controller communications. Monitors are formalised
in terms of a sub-class of finite-state Ligatti et al.’s edit automata [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. A
network composed of n PLCs Ctrli, running in parallel, each of which injected with
a malware Malwi, and enforced by a monitor Moni, is represented as:
      </p>
      <p>
        Mon1 ` fCtrl1 j Malw1g k : : : k Monn ` fCtrln j Malwng :
3 We do not deal with alterations of sensor signals within a PLC, as they can already
occur either at the network level or within the sensor devices [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
Here, the parallel process Ctrli j Malwi is a formal abstraction of the sequential
execution of the PLC code Ctrli injected with the malware Malwi.
      </p>
      <p>
        Then, we propose a synthesis function q y that, given an uncorrupted
(deterministic) PLC Ctrl returns, in polynomial time, a syntactically deterministic
[
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] edit automaton qCtrly to form a monitored PLC that ensures:
observationbased monitoring, transparency, and sound execution of the monitored PLC.
These properties can be expressed with a single algebraic equation:
n
Y qCtrliy ` fCtrli j Malwig '
i=1
n
Y go ` fCtrlig
i=1
(1)
for arbitrary malware Malwi, where ' denotes trace equivalence and go is the
monitor that allows any action. Here, intuitively, each monitor qCtrliy prevents
incorrect executions of the compromised controller Ctrli j Malwi.
      </p>
      <p>
        However, our monitors do not protect against malware that may drag a
monitored PLC into a deadlock state. In fact, Equation 1 does not hold with respect
to weak bisimilarity, which is a notoriously deadlock-sensitive semantic
equivalence. Thus, in order to achieve deadlock-freedom we equip our monitors with the
semantic capability to mitigate those malicious activities that may deadlock the
controller. In practice, our monitors will be able to insert actions, i.e., to emit
correct actions in full autonomy to complete scan cycles. The enforcement
resulting from the introduction of mitigation allows us to recover deadlock-freedom
monitoring by proving Equation 1 with respect to weak bisimilarity.
Outline. Section 2 defines our process calculus to express monitored controllers
injected with malware. Section 3 defines an algorithm to synthesise our monitors.
Section 4 introduces mitigation to recover deadlock-freedom. Section 5 draws
conclusions and discusses related work. In this extended abstract, proofs are
omitted; full proofs can be found in the full version of this paper [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ].
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>A timed process calculus for monitored PLCs</title>
      <p>
        We define our process calculus as an extension of Hennessy and Regan’s TPL [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ].
      </p>
      <p>Let us start with some preliminary notation. We use s; sk 2 Sens for sensor
signals, a; ak 2 Act for actuator commands, and c; ck 2 Chn for channel names.
Controller. In our setting, controllers are nondeterministic sequential timed
processes evolving through three different phases: sensing of sensor signals,
communication with other controllers, and actuation. For convenience, we use four
different syntactic categories to distinguish the four main states of a controller:
Ctrl for initial states, Sens for sensing states, Com for communication states,
and Act for actuation states. In its initial state, a controller is a recursive process
starting its scan cycle in the sensing phase:</p>
      <p>Ctrl 3 P ::= rec X:S
Notice that due to the cyclic behaviour of controllers, the process variable X may
syntactically occur only in the last phase, actuation. We assume time guarded
recursion to avoid undesired zeno behaviours. Intuitively, in time guarded recursion
the process variable must occur prefixed by at least one timed action tick.</p>
      <p>During the sensing phase, the controller waits for a finite number of
admissible sensor signals. If none of those signals arrives in the current time slot then
the controller will timeout moving to the following time slot (we adopt the TPL
construct b c for timeout). The controller may also sleep for a while, waiting for
sensor signals to become stable. The syntax is the following:</p>
      <p>Sens 3 S ::=</p>
      <p>P
b i2I si:SicS
tick:S</p>
      <p>C</p>
      <p>Once the sensing phase is concluded, the controller starts its calculations that
may depend on communications with other controllers. Controllers communicate
to each other for mainly two reasons: either to receive notice about the state of
other physical sub-processes or to require an actuation on a different physical
process that will have an influence on the physical process governed by the
controller. We adopt a channel-based handshake point-to-point communication
paradigm. Notice that, in order to avoid starvation, the communication is always
under timeout. The syntax for the communications phase is:</p>
      <p>Comm 3 C ::=</p>
      <p>P
b i2I ci:CicC
bc:CcC</p>
      <p>A
Thus, our controllers can either listen on a finite number of communication
channels or transmit on specific channels to pass some local information.</p>
      <p>Finally, in the actuation phase the controller eventually transmits a finite
sequence of commands to a number of different actuators, and then, it emits a
special signal end to denote the end of the scan cycle. After that, it restarts its
cycle in the sensing phase via a recursive call denoted with a process variable X.
In order to ensure semantics closure, we also have a construct end:P which will
be only generated at runtime but never used to write PLC programs.</p>
      <p>Act 3 A ::= a:A
end:X</p>
      <p>
        end:P
Remark 1 (Scan cycle duration and maximum cycle limit). Notice that any scan
cycle of a PLC must be completed within a maximum cycle limit which depends
on the controlled physical process; if this time limit is violated the PLC stops
and throws an exception [
        <xref ref-type="bibr" rid="ref29">29</xref>
        ]. Thus, the signal end must occur well before the
maximum cycle limit. We assume that our PLCs successfully complete their
scan cycle in less than half of the maximum cycle limit.
      </p>
      <p>The operational semantics of controllers is given in Table 1. In the
following, we use the metavariables and to range over the set of possible
actions: fs; a; a; c; c; ; tick; endg. These actions denote: sensor readings, actuator
commands, drops of actuator commands, channel transmissions, channel
receptions/drops, internal actions, passage of time, end of a scan cycle, respectively.
Malware. Let us provide a formalisation of the malware code that we assume
may be injected in a controller to compromise its runtime behaviour. The kind
of malware we wish to deal with may perform the following malicious activities:
(i) forging fake channel transmissions towards other controllers (via actions c);
(ii) dropping incoming communications from other controllers (via actions c);
(iii) forging fake actuator commands (via actions a); (iv) dropping actuator
commands launched by the controller (via actions a).</p>
      <p>The formal syntax of the admitted malware is the following:
Malw 3 M
::=</p>
      <p>P
b i2I i:MicM
rec X:M</p>
      <p>X
tick:M
nil
where the prefixes i 2 fc; c; a; ag, for i 2 I, denote the possible malicious actions
mentioned above. Again, we assume time guarded recursion to avoid undesired
zeno behaviours. A straightforward operational semantics is given in Table 2.
Compromised controller. In our setting, a compromised controller may
potentially run in parallel with an arbitrary piece of malware. The syntax is:</p>
      <p>Z ::= P
CCrtl 3 J ::= Z</p>
      <p>S
where Z 2 Ctrl [ Sens [ Comm [ Act denotes a controller in an arbitrary
state, and j is the standard process algebra construct for parallel composition.</p>
      <p>The operational semantics of a compromised controller is given by the
transition rules of Table 3. Rule (Ctrl) models the genuine behaviour of the controller
even in the presence of the malware. Rule (Inject) denotes the injection of a
malicious fabricated action. Rule (DropAct) models the drop of an actuator
command a; thus, the command a never reaches its intended actuator device. Rule
(TimePar) models time synchronisation between the controller and the malware.</p>
      <p>
        We recall that recursion processes in a malware are always time guarded; thus,
a malware can never inject zeno behaviours preventing the passage of time.
Remark 2 (Attacks on channels). Notice that injection/drop on communication
channels affects the interaction between controllers and not within them. For this
reason, we do not have a rule for channels similar to (DropAct). Inter-controller
malicious activities on communication channels will be prevented by the monitor.
Monitored controller(s). The core of our runtime enforcement relies on a
(recursive) timed variant of Ligatti et al.’s edit automata [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ], i.e., a particular class
of automata specifically designed to modify/suppress/insert actions in a generic
system in order to preserve its correct behaviour. Their syntax follows:
Edit 3 E ::= go
      </p>
      <p>Pi2I i= i:Ei
rec X:E</p>
      <p>X
Intuitively, the automaton go will admit any action of the monitored system,
while the edit automaton Pi2I i= i:Ei replaces actions i with i, and then
continues as Ei, for any i 2 I, with I finite. The operational semantics of our
edit automata is the following:
(Go)
go
=
! go
(Edit)
When an edit automaton performs a transition labeled = , with 6= and
= , we say that the automaton suppresses the observable action .</p>
      <p>Our monitored controllers, written E ` fJ g, are constituted by a (potentially)
compromised controller J and an edit automaton E enforcing the behaviour of
J according to the following transition rule for correction/suppression:
(Enforce)</p>
      <p>J
In a monitored controller E ` fJ g with no malware inside, the enforcement never
occurs, i.e., in rule (Enforce) we always have = , and the two components E
and J evolve in a tethered fashion, moving through related correct states.
(ParL)</p>
      <p>N1
N1 k N2
! N10
! N10 k N2</p>
      <p>(ParR)
(TimeSync)
(ChnSync)</p>
      <p>N1
tick
! N10</p>
      <p>N1
c
! N10 N2
N1 k N2</p>
      <p>N2
N1 k N2
! N10 k N20
tick</p>
      <p>! N20
tick</p>
      <p>! N10 k N20</p>
      <p>Obviously, we can easily generalise the concept of monitored controller to a
field communications network of communicating monitored controllers, each one
acting on different actuators. These networks are defined via the grammar:
FNet 3 N ::= E ` fJ g</p>
      <p>
        N k N
and described via the operational semantics given in Table 4. Notice that
monitored controllers may interact with each other via channel communication.
Moreover, they may evolve in time when no communication occurs (we recall that
neither controllers nor malware admit zeno behaviours). This ensures us
maximal progress [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], a desirable time property when modelling real-time systems:
communications are never postponed to future time slots.
      </p>
      <p>Behavioural equalities. In the paper, we adopt standard behavioural equivalences
between (networks of) monitored controllers. In particular, we use trace
equivalence, written ', weak similarity, denoted v, and weak bisimilarity, written .
2.1</p>
      <p>Use case: a small water-tank system
In this section, we specify the controller of a simple water-tank system depicted
in Figure 1. Basically, in this system the water is pumped into the tank via a
pump. Furthermore, a valve connects the tank with an external unit which is
not represented. Here, we assume that the incoming water flow is lower than the
out-coming flow passing through the valve.</p>
      <p>The PLC works as follows: it waits for one time slot (to get stable sensor
signals) and then checks the water level of the tank, distinguishing between
three possible states. If the tank reaches a low level (signal l) then the pump is
turned on (command on) and the valve gets closed (command close). If the level of
the tank is high (signal h) then the PLC listens for requests arriving at channel
open_req to open the valve; if the PLC gets a request then it opens the valve
(command open) and returns; otherwise, it timeouts and then turns the pump off
(commands o ) and closes the valve (command close). Finally, if the tank is at
some intermediate level between l and h (signal m) the PLC listens for requests
of water; if it gets a request of water (via the channel open_req) then it turns the
pump on and opens the valve, letting the water flow out; otherwise, if it receives</p>
      <p>ú
pump
water
on, open
off, close</p>
      <p>Tank
h−
m−.</p>
      <p>..
ml−−.. ./</p>
      <p>. valve
PLC</p>
      <p>l, m, h
open req, close req
a request to close the valve (via the channel close_req) then it closes the valve,
and then returns.</p>
      <p>The specification of the controller PLC mentioned above is the following:
rec X: tick:bl:on:close:end:X + h:bopen_req:open:end:Xc(o :close:end:X)c(end:X)</p>
      <p>+ m:bopen_req:on:open:end:X + close_req:close:end:Xc(end:X) :
3</p>
    </sec>
    <sec id="sec-3">
      <title>Synthesis of monitoring proxies</title>
      <p>In Table 5, we provide a synthesis function q y that given a controller P 2 Ctrl
returns a syntactically deterministic edit automaton E 2 Edit enforcing the
correct behaviour of P , independently of the presence of an arbitrary malware
M 2 Malw that attempts to inject and/or drop both actuator commands and
channel communications.</p>
      <p>In our synthesis, we adopt the following notation for co-actions regarding
actuator commands and channel communications: Act , fa j a 2 Actg and
Chn , fc j c 2 Chng. Furthermore, we set Act , Act[Act and Chn , Chn[Chn.</p>
      <p>Let us comment on the details of the synthesis function q y of Table 5.
The edit automaton associated to listening on sensor signals allows all incoming
signals expected by the controller, together with the passage of time due to
eventual timeouts. All other actions are suppressed. The edit automaton associated to
the listening on communication channels is similar, except that communications
that are not admitted by the controller are suppressed to prevent both drops and
injections on system channels, as well as, covert communications between
colluding malware running in different PLCs. Channel transmissions are allowed only
when occurring, in the right order, on those channels intended by the controller;
all other actions are suppressed. Only genuine actuator commands (again, in the
, rec X:qSy
qbPi2I si:SicSy , rec Y: Pi2I si=si:qSiy + tick=tick:qSy +
, rec Y: tick=tick:qSy +</p>
      <p>P
= :Y
qbc:C1cC2y
qbPi2I ci:CicCy , rec Y: Pi2I ci=ci:qC2iAyct+[tCichkn=tick:qCy + P
2Act
, rec Y: c=c:qC1y + tick=tick:qC2y + = :Y +
, rec Y: a=a:qAy + = :Y +
, rec Y: end=end:X +</p>
      <p>P
2Act [Chn</p>
      <p>P
2Act</p>
      <p>P
2Act nfa;ag
= :Y</p>
      <p>P
2Act [Chn</p>
      <p>= :Y
= :Y +
= :Y +</p>
      <p>P
2Chn nfcg
P = :Y
2Chn</p>
      <p>P
2Chn n[i2I fcig
= :Y
= :Y
qrec X:Sy
qtick:Sy
qa:Ay
qend:Xy
right order) are allowed. Drops of actuator commands, the only possible
intracontroller interaction occurring between the genuine controller and the malware,
are allowed because we want an observation-based monitoring. Finally, the
monitoring edit automaton and the associated controller do synchronise at the end of
each controller cycle via the action end: all other actions emitted by the
compromised controller are suppressed, included those actions coming from the genuine
controller that was left behind in its execution due to some injection attack
mimicking (part of) some correct behaviour. We recall that only the construct end:X
(and not end:P ) is used to write PLC programs.</p>
      <p>Let us start with two easy observations.</p>
      <p>Remark 3 (Observation-based monitoring). The edit automata resulting from
our synthesis never correct -actions (i.e., non-observable actions).
Remark 4 (Colluding malicious activities). Any inter-controller activity which
does not comply with the genuine behaviour of the PLC under scrutiny is
suppressed by the enforcement.</p>
      <p>
        The synthesis proposed in Table 5 is suitable for implementation.
Proposition 1 (Determinism preservation). Let P 2 Ctrl be a
deterministic controller. The automaton qP y is syntactically deterministic in the sense of [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
Furthermore, our synthesis algorithm is computationally feasible. The
complexity of the synthesis is quadratic on the dimension of the controller, where,
intuitively, the dimension of a controller P 2 Ctrl, written dim(P ), is given by the
number of prefixes 2 Act [ Chn [ Sens [ ftick; endg occurring in it (see [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]).
Proposition 2 (Polynomial complexity). Let P 2 Ctrl be a deterministic
controller, the complexity to synthesise qP y is O(n2), with n = dim(P ).
      </p>
      <p>As required at the beginning of this section, the synthesised edit automata
are always transparent, i.e., they never introduce non-genuine behaviours.
Proposition 3 (Transparency). If P 2 Ctrl then qP y ` fP g
go ` fP g:</p>
      <p>Furthermore, our enforcement enjoys soundness preservation : in a monitored
controller, a malware may never trigger an incorrect behaviour.</p>
      <p>Proposition 4 (Soundness). Let P be an arbitrary controller and M be an
arbitrary malware. Then, qP y ` fP j M g v qP y ` fP g.</p>
      <p>In the next proposition, we provide a result that is somehow complementary
to Proposition 4. The intuition being that in a monitored controller qP y ` fP j
M g the controller P may execute all its (genuine) execution traces even in the
presence of an arbitrary malware M . Said in other words, the controller P has a
chance to follow (and complete) its correct execution, even when compromised
by the presence of a malware M .</p>
      <p>Proposition 5. Let P be an arbitrary controller and M be an arbitrary
malware. Then, qP y ` fP j M g w qP y ` fP g.</p>
      <p>By applications of Propositions 3, 4, and 5 we can summarise our enforcement
in a single equation.</p>
      <p>Theorem 1 (Weak enforcement). Let P 2 Ctrl be an arbitrary controller
and M 2 Malw be an arbitrary malware. Then, qP y ` fP j M g ' go ` fP g :</p>
      <p>The result of weak enforcement scales to field communications networks of
communicating controllers compromised by the presence of colluding malware.
Proposition 6 (Weak enforcement of field networks). Let Pi 2 Ctrl and
Mi 2 Malw, for 1 i n. Then, Qin=1 qPiy ` fPi j Mig ' Qin=1 go ` fPig :</p>
      <p>However, our enforcement does not enjoy deadlock-freedom.</p>
      <p>Remark 5 (Injection attacks may prevent deadlock-freedom). In a monitored
controller of the form qP y ` fP j M g, it may well happen that the malware
M misleads the edit automaton qP y by injecting an untimed trace M 1!
: : : n! M 0 of actions, with i 6= tick, compatible with the correct behaviour
of the controller, in the sense that the very same trace may be executed by
P : P 1! : : : n! Q, for some state Q. This would give rise to the
following admissible execution trace for the monitored controller: qP y ` fP j
M g 1! : : : n! qQy ` fP j M 0g, in which the actual controller P
remains inactive. At that point, if the malware M 0 stops following an admissible
runtime behaviour for the controller, the edit automaton qQy will suppress all
possible actions, even those proposed by P , which was left behind in its
execution. Thus, the monitored controller will continue its evolution as follows:
qQy ` fP j M 0g ! : : : ! qQy ` fP 0 j M 00g. In this case, as neither the
controller nor the malware can give rise to zeno behaviours, the enforced system
may eventually reach a deadlock state in which (i) P 0 = end:X, (ii) M 00 = tick:M 000,
for some M 000, or M 00 = nil, and (iii) qQy does not allow end-actions because it
requires some actions (e.g., actuations) to be performed before ending the
controller cycle.
Notice that Remark 5 is not in contradiction with Proposition 5 because in that
proposition we proved that a controller has a chance to follow and complete
its correct behaviour in the presence of an arbitrary malware. Here, we say a
different thing: a malware has a chance to deadlock our monitored controllers.
4</p>
    </sec>
    <sec id="sec-4">
      <title>Mitigation: the recipe for deadlock-freedom</title>
      <p>In this section, we introduce an extra transition rule for monitored controllers
to implement mitigation, i.e., the insertion of activities in full autonomy, when
the controller has lost contact with its enforcer:
(Mitigation)</p>
      <p>J
end
Intuitively, if the compromised controller signals the end of the scan cycle by
emitting the action end and, at the same time, the current edit automaton E is
not in the same state, then E will command the execution of a safe trace, without
any involvement of the controller, to reach the end of the controller cycle. When
both the controller and the edit automaton will be aligned (at the end of the
cycle) they will synchronise on the action end, via an application of the transition
rule (Enforce), and from then on they will continue in a tethered fashion.</p>
      <p>Notice that in a monitored controller E ` fJ g where J is corrupted by some
malware, the two components E and J may get misaligned as they may reach
unrelated states. For instance, in case of drop of actuator commands the corrupted
controller J may reach an incorrect state, leaving behind its monitoring edit
automata E. In this case, the remaining observable actions in the current cycle
will be suppressed until the controller reaches the end of the scan cycle, signalled
by the emission of an end-action (notice that since our malware are time-guarded
they cannot introduce zeno behaviours to prevent a controller to reach the end
of its scan cycle). Once the compromised controller has been driven to the end
of its cycle, the transition rule (Mitigation) goes into action.</p>
      <p>Remark 6. The assumption made in Remark 1 ensures us enough time to
complete the mitigation of the scan cycle, well before the maximum cycle limit.</p>
      <p>As a main result, we prove that with the introduction of the rule (Mitigation)
our runtime enforcement for controllers works faithfully up to weak bisimilarity,
ensuring deadlock-freedom.</p>
      <p>Theorem 2 (Strong enforcement). Let P 2 Ctrl be an arbitrary controller
and M 2 Malw be an arbitrary malware. Then, qP y ` fP j M g go ` fP g :</p>
      <p>Strong enforcement easily scales to field networks of communicating
controllers compromised by the presence of (potentially) colluding malware.
Corollary 1 (Strong enforcement of field networks). LQetnPi 2 Ctrl and
Mi 2 Malw, for 1 i n. Then, Qin=1 qPiy ` fPi j Mig i=1 go ` fPig :</p>
    </sec>
    <sec id="sec-5">
      <title>Conclusions and related work</title>
      <p>
        We have defined a formal language to express networks of monitored PLCs,
potentially compromised with colluding malware that may forge/drop actuator
commands and inter-controller communications. We do not deal with alterations
of sensor signals within a PLC, as they can already occur either at the network
level or within the sensor devices [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. The runtime enforcement has been achieved
via a finite-state sub-class of Ligatti’s edit automata equipped with an ad-hoc
operational semantics to deal with system mitigation, by inserting actions in full
autonomy when the monitored controller is not able to do so in a correct manner.
Then, we have provided a synthesis algorithm that, given a deterministic
uncorrupted controller, returns, in polynomial time, a syntactically deterministic edit
automata to enforce the correctness of the controller. The proposed enforcement
meets a number of requirements: observation-based monitoring, transparency,
soundness, and deadlock-freedom.
      </p>
      <p>
        Related work. The notion of runtime enforcement was introduced by
Schneider [
        <xref ref-type="bibr" rid="ref28">28</xref>
        ] to enforce security policies. These properties are enforced by means of
security automata, a kind of automata that terminates the monitored system in
case of violation of the property. Ligatti et al. [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] extended Schneider’s work
by proposing the notion of edit automaton, i.e., an enforcement mechanism able
of replacing, suppressing, or even inserting system actions. In general, Ligatti et
al.’s edit automata have an enumerable number of states, whereas in the
current paper we restrict ourselves to finite-state edit automata. Furthermore, in
its original definition the insertion of actions is possible at any moment, whereas
our monitoring edit automata can insert actions, via the rule (Mitigation), only
when the PLC under scrutiny reaches a specific state, i.e., the end of the scan
cycle. Finally, our actions of the form = can be easily expressed in the
original formulation by inserting the action and then suppressing the action .
Unlike Schneider and Ligatti et al., we do not enforce specific properties for all
admissible systems (in our case, controllers) but we ensure the preservation of
the correct semantics of a corrupted controller. Bielova [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] provided a stronger
notion of enforceability by introducing a predictability criterion to prevent
monitors from transforming invalid executions in an arbitrary manner. Falcone et
al. [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] proposed a synthesis algorithm, relying on Street automata, to translate
most of the property classes defined within the Safety-Progress hierarchy [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ]
into enforcers. Könighofer et al. [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] proposed a synthesis algorithm that given
a safety property returns a monitor, called shield, that analyses outputs of
reactive systems. More recently, Pinisetty et al. [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ] have proposed a bi-directional
runtime enforcement mechanism for reactive systems, and more generally for
cyber-physical systems, to correct both inputs and outputs. Aceto et al. [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]
developed an operational framework to enforce safety properties expressed in HML
logic with recursion ( HML) by relying on suppression only. Enforceability of
modal -calculus (a reformulation of HML) was previously tackled by
Martinelli and Matteucci [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ]. More recently, Cassar [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] defined a general framework
to compare different enforcement models and different correctness criteria,
including optimality.
      </p>
      <p>
        Finally, in our companion paper [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] we have abstracted over PLC
implementations and provided a simple language of regular properties to express
correctness properties that should be enforced upon completion of PLC scan cycles.
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Abadi</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Blanchet</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fournet</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          : The Applied Pi Calculus: Mobile Values, New Names, and
          <article-title>Secure Communication</article-title>
          .
          <source>Journal of the ACM</source>
          <volume>65</volume>
          (
          <issue>1</issue>
          ), 1:
          <fpage>1</fpage>
          -
          <lpage>1</lpage>
          :
          <fpage>41</fpage>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Abbasi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hashemi</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>Ghost in the PLC designing an undetectable orogrammable logic controller rootkit via pin control attack</article-title>
          .
          <source>In: Black Hat</source>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Aceto</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Achilleos</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Francalanza</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ingólfsdóttir</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kjartansson</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          Ö.:
          <article-title>On the Complexity of Determinizing Monitors</article-title>
          .
          <source>In: CIAA. LNCS</source>
          , vol.
          <volume>10329</volume>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>13</lpage>
          . Springer (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Aceto</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cassar</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Francalanza</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ingólfsdóttir</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>On runtime enforcement via suppressions</article-title>
          .
          <source>In: CONCUR</source>
          . pp.
          <volume>34</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>34</lpage>
          :
          <fpage>17</fpage>
          .
          <string-name>
            <surname>Schloss Dagstuhl</surname>
          </string-name>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Bielova</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A theory of constructive and predictable runtime enforcement mechanisms</article-title>
          .
          <source>Ph.D. thesis</source>
          , University of Trento (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Cardelli</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gordon</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Mobile ambients</article-title>
          .
          <source>TCS</source>
          <volume>240</volume>
          (
          <issue>1</issue>
          ),
          <fpage>177</fpage>
          -
          <lpage>213</lpage>
          (
          <year>2000</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Cassar</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          :
          <article-title>Developing Theoretical Foundations for Runtime Enforcement</article-title>
          .
          <source>Ph.D. thesis</source>
          , University of Malta and Reykjavik University (
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Falcone</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mounier</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fernandez</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Richier</surname>
          </string-name>
          , J.:
          <article-title>Runtime enforcement monitors: composition, synthesis, and enforcement abilities</article-title>
          .
          <source>FMSD</source>
          <volume>38</volume>
          (
          <issue>3</issue>
          ),
          <fpage>223</fpage>
          -
          <lpage>262</lpage>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Giraldo</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Urbina</surname>
            ,
            <given-names>D.I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cardenas</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Valente</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Faisal</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ruths</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tippenhauer</surname>
            ,
            <given-names>N.O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sandberg</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Candell</surname>
          </string-name>
          , R.:
          <source>A Survey of Physics-Based Attack Detection in Cyber-Physical Systems. ACM Comput. Surv</source>
          .
          <volume>51</volume>
          (
          <issue>4</issue>
          ),
          <volume>76</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>76</lpage>
          :
          <fpage>36</fpage>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Hennessy</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rathke</surname>
          </string-name>
          , J.:
          <article-title>Towards a behavioural theory of access and mobility control in distributed systems</article-title>
          .
          <source>TCS</source>
          <volume>322</volume>
          (
          <issue>3</issue>
          ),
          <fpage>615</fpage>
          -
          <lpage>669</lpage>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Hennessy</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Regan</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          :
          <article-title>A process algebra for timed systems</article-title>
          .
          <source>Information and Computation</source>
          <volume>117</volume>
          (
          <issue>2</issue>
          ),
          <fpage>221</fpage>
          -
          <lpage>239</lpage>
          (
          <year>1995</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Könighofer</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Alshiekh</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bloem</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Humphrey</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Könighofer</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Topcu</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Shield synthesis</article-title>
          .
          <source>FMSD</source>
          <volume>51</volume>
          (
          <issue>2</issue>
          ),
          <fpage>332</fpage>
          -
          <lpage>361</lpage>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Lanese</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bedogni</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Di</surname>
            <given-names>Felice</given-names>
          </string-name>
          ,
          <string-name>
            <surname>M.</surname>
          </string-name>
          :
          <article-title>Internet of Things: a process calculus approach</article-title>
          .
          <source>In: ACM SAC</source>
          . pp.
          <fpage>1339</fpage>
          -
          <lpage>1346</lpage>
          . ACM (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Lanotte</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A Calculus of Cyber-Physical Systems</article-title>
          .
          <source>In: LATA. LNCS</source>
          , vol.
          <volume>10168</volume>
          , pp.
          <fpage>115</fpage>
          -
          <lpage>127</lpage>
          . Springer (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Lanotte</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A semantic theory of the Internet of Things</article-title>
          .
          <source>Information and Computation</source>
          <volume>259</volume>
          (
          <issue>1</issue>
          ),
          <fpage>72</fpage>
          -
          <lpage>101</lpage>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Lanotte</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Munteanu</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>A process calculus approach to correcteness enforcement of PLCs (full version)</article-title>
          .
          <source>CoRR abs/2007</source>
          .09399 (
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Lanotte</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Munteanu</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Runtime Enforcement for Control System Security</article-title>
          . In: CSF. pp.
          <fpage>246</fpage>
          -
          <lpage>261</lpage>
          . IEEE (
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Lanotte</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Munteanu</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Viganò</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>A Formal Approach to Physicsbased Attacks in Cyber-physical Systems</article-title>
          .
          <source>ACM TOPS 23(1)</source>
          , 3:
          <fpage>1</fpage>
          -
          <lpage>3</lpage>
          :
          <fpage>41</fpage>
          (
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Lanotte</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tini</surname>
            ,
            <given-names>S.:</given-names>
          </string-name>
          <article-title>A Probabilistic Calculus of Cyber-Physical Systems</article-title>
          .
          <source>Information and Computation</source>
          <volume>104618</volume>
          ,
          <fpage>1</fpage>
          -
          <lpage>30</lpage>
          (
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Ligatti</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bauer</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Walker</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Edit automata: enforcement mechanisms for run-time security policies</article-title>
          .
          <source>Int. J. Inf. Sec</source>
          .
          <volume>4</volume>
          (
          <issue>1-2</issue>
          ),
          <fpage>2</fpage>
          -
          <lpage>16</lpage>
          (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Macedonio</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Merro</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A semantic analysis of key management protocols for wireless sensor networks</article-title>
          .
          <source>Science of Computer Programming</source>
          <volume>81</volume>
          ,
          <fpage>53</fpage>
          -
          <lpage>78</lpage>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Manna</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pnueli</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>A Hierarchy of Temporal Properties</article-title>
          .
          <source>Tech. rep.</source>
          , Stanford University (
          <year>1987</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Martinelli</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matteucci</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          :
          <article-title>Through modeling to synthesis of security automata</article-title>
          .
          <source>ENTCS 179</source>
          ,
          <fpage>31</fpage>
          -
          <lpage>46</lpage>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>McLaughlin</surname>
            ,
            <given-names>S.E.</given-names>
          </string-name>
          :
          <article-title>CPS: stateful policy enforcement for control system device usage</article-title>
          .
          <source>In: ACSAC</source>
          . pp.
          <fpage>109</fpage>
          -
          <lpage>118</lpage>
          . ACM (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Mohan</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bak</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Betti</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yun</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sha</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Caccamo</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>S3A: secure system simplex architecture for enhanced security and robustness of cyber-physical systems</article-title>
          .
          <source>In: HiCoNS</source>
          . pp.
          <fpage>65</fpage>
          -
          <lpage>74</lpage>
          . ACM (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Pinisetty</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Roop</surname>
            ,
            <given-names>P.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Smyth</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Allen</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tripakis</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hanxleden</surname>
          </string-name>
          , R.:
          <article-title>Runtime enforcement of cyber-physical systems</article-title>
          .
          <source>ACM TECS 16(5s)</source>
          ,
          <volume>178</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>178</lpage>
          :
          <fpage>25</fpage>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27.
          <string-name>
            <surname>Radvanovsky</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <source>Project shine: 1</source>
          ,
          <issue>000</issue>
          ,
          <article-title>000 internet-connected SCADA and ICS stystems and counting (</article-title>
          <year>2013</year>
          ), Tofino Security
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          28.
          <string-name>
            <surname>Schneider</surname>
            ,
            <given-names>F.B.</given-names>
          </string-name>
          :
          <article-title>Enforceable security policies</article-title>
          .
          <source>ACM TISSEC 3</source>
          (
          <issue>1</issue>
          ),
          <fpage>30</fpage>
          -
          <lpage>50</lpage>
          (
          <year>2000</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          29.
          <string-name>
            <surname>Spenneberg</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brüggerman</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schwartke</surname>
          </string-name>
          , H.:
          <article-title>PLC-Blaster: A Worm Living Solely in the PLC</article-title>
          . In: Black Hat. pp.
          <fpage>1</fpage>
          -
          <lpage>16</lpage>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>