<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>Workshop on Artificial Intelligence and Formal Verification, Logics, Automata and Synthesis (OVERLAY),
Rende, Italy, November</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Pushing Runtime Verification to the Limit: May Process Semantics Be With Us∗</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Dario Della Monica</string-name>
          <email>1dario.dellamonica@uniud.it</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Adrian Francalanza</string-name>
          <email>2adrian.francalanza@um.edu.mt</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>University of Malta</institution>
          ,
          <country country="MT">Malta</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>University of Udine</institution>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <volume>1</volume>
      <fpage>9</fpage>
      <lpage>20</lpage>
      <abstract>
        <p>We propose a combined approach that permits automated formal verification to be spread across the pre- and post-deployment phases of a system development, with the aim of calibrating the management of the verification burden. Our approach combines standard model checking methods with runtime verification, a relatively novel formal technique that verifies a system during its execution. We carry out our study in terms of the Hennessy-Milner Logic, a branching-time logic for specifying reactive system correctness. Whereas we will be mainly concerned with limiting the model checking verification burden, runtime verification has been shown to handle a strict subset of the expressible properties in our logic of study, posing constraints on what can be shifted to the post-deployment phase. We present a solution, based on modal transition systems and modal refinement, for the fragment of the Hennessy-Milner Logic devoid of recursion, i.e., without least and greatest fixpoint operators.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        Introduction Model checking (MC) [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] is a widely accepted pre-deployment verification technique that
checks whether a system satisfies or violates a property by potentially analysing all the possible system
behaviours. By contrast, runtime verification (RV) [
        <xref ref-type="bibr" rid="ref11 ref30">30, 11</xref>
        ] is a lightweight verification technique aimed
at mitigating scalability issues, such as the state explosion problem, typically associated with traditional
verification techniques like MC. RV attempts to infer the satisfaction (or violation) of a correctness
property from the analysis of the current execution of the system under scrutiny using monitors [
        <xref ref-type="bibr" rid="ref22 ref23">22, 23</xref>
        ]. It
is thus performed post-deployment (on actual system execution), which is appealing for component-based
applications (parts of which may not be available for analysis pre-deployment), as well as for dynamic
settings such as mobile computing (where components are downloaded and installed at runtime). The
technique has fostered a number of verification tools, e.g., [
        <xref ref-type="bibr" rid="ref10 ref18 ref20 ref27 ref29 ref32 ref34 ref7 ref8 ref9">9, 10, 18, 20, 27, 29, 32, 7, 8, 34</xref>
        ] to name but
a few, and has proved effective in various real-world scenarios [
        <xref ref-type="bibr" rid="ref14 ref21 ref37">14, 37, 21</xref>
        ].
      </p>
      <p>
        Despite its advantages, RV is limited when compared to verification techniques such as MC because
certain correctness properties cannot be verified at runtime [
        <xref ref-type="bibr" rid="ref16 ref2 ref26 ref3 ref31 ref33 ref4">31, 33, 16, 26, 2, 3, 4</xref>
        ]. For instance, MC
makes it possible to check for both safety and liveness properties, by providing either a positive or a
negative answer, according to whether the system conforms with the specifications; RV, on the other
hand, can only return a positive verdict for certain liveness properties (called co-safety properties [
        <xref ref-type="bibr" rid="ref15 ref4">15, 4</xref>
        ])
or a negative one for safety conditions. Moreover, RV induces a runtime overhead over the execution of a
monitored system, which should ideally be kept to a minimum [
        <xref ref-type="bibr" rid="ref11 ref30">30, 11</xref>
        ].
      </p>
      <p>Monitorable Fragments
ϕ, φ ∈ μHML ::= tt
| ϕ ∨ φ
| hαiϕ
| min X.ϕ
| X
(truth)
(disjunction)
(possibility)
(min. fixpoint)
(rec. variable)</p>
      <p>Jtt, ρK
Jϕ1∨ϕ2, ρK</p>
      <p>Jhαiϕ, ρK
JminX.ϕ, ρK</p>
      <p>JX, ρK
d=ef Sta
ddeeff Jϕ1, ρK ∪ Jϕ2, ρK
=
= s | ∃r.s −α→ r and r ∈ Jϕ, ρK
d=ef T {S ∈ Sta | Jϕ, ρ[X 7→ S]K ⊆ S}
d=ef ρ(X)
| ff
| ϕ ∧ φ
| [α]ϕ
| max X.ϕ
(falsehood)
(conjunction)
(necessity)
(max. fixpoint)</p>
      <p>Jff, ρK
Jϕ1∧ϕ2, ρK</p>
      <p>
        J[α]ϕ, ρK
JmaxX.ϕ, ρK
def
= ∅
ddeeff Jϕ1, ρK ∩ Jϕ2, ρK
=
= s | ∀r.s −α→ r implies r ∈ Jϕ, ρK
d=ef S {S ∈ Sta | S ⊆ Jϕ, ρ[X 7→ S]K}
θ, ϑ ∈ sHML ::= tt
π, $ ∈ cHML ::= tt
| ff
| ff
| [α]θ
| hαiπ
| θ∧ϑ
| π∨$
| max X.θ
| min X.π
| X
| X
Syntax
Semantics
Hennessy-Milner Logic with Recursion (μHML) RV’s limits in terms of verifiable properties is
evidenced more for branching-time logics, that are able to express properties describing behaviour over
multiple system executions. In recent work [
        <xref ref-type="bibr" rid="ref1 ref25 ref26 ref3">25, 26, 1, 3</xref>
        ], one such branching-time logic called μHML
[
        <xref ref-type="bibr" rid="ref13 ref5">13, 5</xref>
        ] is studied from an RV perspective. Figure 1 outlines the syntax of the logic μHML, along with its
semantics, defined over a Labelled Transition System (LTS), i.e., triples hSta, Act, −−→i consisting of a
set of states s, r ∈ Sta, a set of actions α ∈ Act, and a transition relation between states labelled by
actions, s −→α r; as in [
        <xref ref-type="bibr" rid="ref26 ref5">5, 26</xref>
        ], the semantic definition employs an environment from μHML logical variables,
Vars, to sets of states, ρ ∈ (Vars * P(Sta)) (see Figure 1). One of the main contributions of [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ] is the
identification of an expressively maximal, runtime-verifiable subset of the logic, reported in Figure 1 as the
grammar for sHML and cHML (see also [
        <xref ref-type="bibr" rid="ref24 ref26">26, 24</xref>
        ]); the authors show how these classes provide an easy
syntactic check for determining whether a property satisfaction (or violation) can be determined using the
RV technique. The cHML (resp., sHML) fragment of μHML is said to be positively monitorable (resp.,
negatively monitorable) [
        <xref ref-type="bibr" rid="ref26 ref4">26, 4</xref>
        ].
      </p>
      <p>
        Extending the applicability of monitoring towards a combined verification We build on the
findings of [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ], with the aim of extending the applicability of RV to a larger class of μHML properties
other than sHML ∪ cHML from Figure 1. Specifically, we propose a combined approach that permits
automated formal verification to be spread across the pre- and post-deployment phases of a system
development, with the aim of calibrating the management of the verification burden while combining the
strengths of MC with those of RV. As an illustrative example, consider the μHML property (1) below,
describing systems that can perform action a, prefix hai(. . .), and reach a state from where they can either
perform action b, subformula hbitt, or else can never perform action c, subformula [c]ff.
(1)
(2)
According to Figure 1, (1) turns out not to be runtime-verifiable because of the subformula [c]ff; intuitively,
whereas a system execution exhibiting action a followed by action b suffices to prove that the system satisfies
(1), an RV monitor cannot determine whether a system can never produce action c after performing action
a from the observation of only a single system execution [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ]. However, property (1) can be expressed as
the (logically equivalent) formula
      </p>
      <p>
        (haihbitt) ∨ (hai[c]ff)
whereby we note that the subformula haihbitt is runtime verifiable, according to [
        <xref ref-type="bibr" rid="ref25 ref26 ref3">25, 26, 3</xref>
        ]. We argue that
reformulations such as (2) allow for a combined approach to verification, where part of the property, e.g.,
the (smaller) subformula hai[c]ff, can be checked prior system deployment using MC, and the remaining
part of the property, e.g., haihbitt, can be runtime-verified during system execution.
      </p>
      <p>We therefore aim to devise general analysis techniques that reformulate any μHML formula into
either conjunctions or disjunctions, i.e., ϕRV ∧ ϕMC or ϕRV ∨ ϕMC, where ϕRV and ϕMC denote the
runtime-verifiable and model-checkable formula components, respectively. From a software engineering
perspective, we envisage at least two ways how this decomposition between pre- and post-deployment
verification can be fruitful:
1. The ensuing combined approach may be used as a means to minimise the verification effort required
prior to the deployment of a system. E.g., in the case of (2), the model-checked subformula
ϕMC = hai[c]ff is smaller than the full formula (1), since we would be offloading a degree of
verification onto the runtime phase when runtime-verifying for ϕRV = haihbitt. Moreover, for
disjunction decompositions such as (2), the satisfaction of ϕMC prior to deployment obviates the
need for any runtime analysis, minimising runtime overheads (a dual argument applies for conjunction
decompositions and ϕMC violations).
2. In settings where software correctness is desirable but not essential, a combined approach can be
used as a means to circumvent full-blown MC. Specifically, instead of model-checking for (1), a
system may be runtime-verified for ϕRV = haihbitt during its pilot launch, acting as a vetting phase:
if ϕRV is satisfied during RV, this means that, by (2), (1) is satisfied as well; if not, we then proceed
to model-check the system offline wrt. ϕMC = hai[c]ff.</p>
      <p>
        A partial solution based on modal transition systems and modal refinement From a technical
point of view, the problem amounts, to computing the maximal monitorable semantic fragment of a given
μHML formula ϕ, that is, the formula ψ such that
• ψ ∈ cHML (ψ is positively monitorable),
• JψK ⊆ JϕK (every process that satisfies ψ also satisfies ϕ, i.e, ψ is a semantic fragment of ϕ),
• for every ψ0 ∈ cHML, we have that Jψ0K ⊆ J K ψ (ψ is maximal)
ϕ implies Jψ0K ⊆ J K
Dually, for the case of negatively monitorable formulas (fragment sHML), we are interested in the minimal
monitorable formula of which ϕ is a semantic fragment; we focus on the former formulation only since the
latter one can be solved by exploiting the duality between the two fragments (as in [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]).
      </p>
      <p>
        Instead of trying to obtain the desired formula through syntactic transformations, we adopt a semantic
approach that proved itself successful for the logic HML, i.e., the fragment of μHML devoid of fixpoint
operators. We first transform the input formula into a modal transition system (MTS ) [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], which can be
thought of as graphical representations of formulas: MTSs suit our purpose particularly well because they
are amenable to manipulations while preserving the information about the meaning of the formula. By
using this technique, we can inherit known results from concurrency theory, such as the characterization
and classification of several process semantics [
        <xref ref-type="bibr" rid="ref19 ref35 ref36">19, 35, 36</xref>
        ]. In particular, back and forth translations
between HML formulas and MTSs are known [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], and a preorder over MTSs, called modal refinement, has
been defined that captures the semantic relationship between HML specifications: an MTS M1 precedes
another MTS M2 in this modal refinement preorder whenever the set of processes that satisfy ϕM1 (the
translation of M1 into an HML formula) contains set of processes that satisfy ϕM2 . Using such results we
are able to identify the class of MTSs corresponding to the monitorable fragment of HML and to single out
the MTS corresponding to the maximal monitorable semantic fragment of a given HML formula. Then,
by employing the translation from MTSs back to HML formulas, we obtain the monitorable specification
that we are looking for.
      </p>
      <p>
        Future directions Extending our approach to the full μHML remains an open issue: for this purpose,
MTSs should be extended with cycles so as to enable them to “mimic” fixpoint operators, which somehow
correspond to recursion. Once this obstacle is solved, we can also investigate the application of our
methods to the linear-time setting, where there are still formulas that are not monitorable [
        <xref ref-type="bibr" rid="ref3 ref4">3, 4</xref>
        ].
      </p>
      <p>
        Another direction we intend to pursue is that of extending our techniques to settings with enriched
monitoring capabilities. A number of these settings have recently been investigated for the logic μHML
in the work [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] by considering monitoring setups with the ability to recognize when a process terminates,
or the ability to infer the possible (1-step) actions from a specific state (even though the computation will
then continue executing along only one of these actions). Using some of the aforementioned results from
the field of concurrency theory and process semantics, our approach should extend in a straightforward
manner to cope with the enhanced monitoring capabilities. More importantly, this study marries well
with our aims for a multi-pronged verification methodology along the lines advocated in [
        <xref ref-type="bibr" rid="ref21 ref28 ref6">6, 21, 28</xref>
        ].
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Achilleos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          .
          <article-title>Monitoring for silent actions</article-title>
          . In S. Lokam and R. Ramanujam, editors,
          <source>FSTTCS</source>
          , volume
          <volume>93</volume>
          <source>of LIPIcs</source>
          , pages
          <volume>7</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>7</lpage>
          :
          <fpage>14</fpage>
          ,
          <string-name>
            <surname>Dagstuhl</surname>
          </string-name>
          , Germany,
          <year>2017</year>
          .
          <article-title>Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Achilleos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          .
          <article-title>A Framework for Parameterized Monitorability</article-title>
          . In C. Baier and U. D. Lago, editors,
          <source>Proc. of the 21st International Conference on Foundations of Software Science and Computation Structures (FOSSACS)</source>
          , volume
          <volume>10803</volume>
          <source>of LNCS</source>
          , pages
          <fpage>203</fpage>
          -
          <lpage>220</lpage>
          . Springer,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Achilleos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          , and
          <string-name>
            <given-names>K.</given-names>
            <surname>Lehtinen</surname>
          </string-name>
          .
          <article-title>Adventures in monitorability: From branching to linear time and back again</article-title>
          .
          <source>Proceedings of the ACM on Programming Languages</source>
          ,
          <volume>3</volume>
          (POPL):
          <volume>52</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>52</lpage>
          :
          <fpage>29</fpage>
          ,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Achilleos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          , and
          <string-name>
            <given-names>K.</given-names>
            <surname>Lehtinen</surname>
          </string-name>
          .
          <article-title>An operational guide to monitorability</article-title>
          .
          <source>In Software Engineering and Formal Methods - 17th International Conference, SEFM 2019</source>
          , Oslo, Norway,
          <source>September 18-20</source>
          ,
          <year>2019</year>
          , Proceedings, volume
          <volume>11724</volume>
          <source>of LNCS</source>
          , pages
          <fpage>433</fpage>
          -
          <lpage>453</lpage>
          . Springer,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. G.</given-names>
            <surname>Larsen</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Srba</surname>
          </string-name>
          .
          <source>Reactive Systems: Modelling, Specification and Verification</source>
          . Cambridge Univ. Press,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>C.</given-names>
            <surname>Artho</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Barringer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Goldberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Havelund</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Khurshid</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Lowry</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Pasareanu</surname>
          </string-name>
          , G. Roşu,
          <string-name>
            <given-names>K.</given-names>
            <surname>Sen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Visser</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R.</given-names>
            <surname>Washington</surname>
          </string-name>
          .
          <article-title>Combining test case generation and runtime verification</article-title>
          .
          <source>Theoretical Computer Science</source>
          ,
          <volume>336</volume>
          (
          <issue>2</issue>
          ):
          <fpage>209</fpage>
          -
          <lpage>234</lpage>
          ,
          <year>2005</year>
          .
          <article-title>Abstract State Machines and High-Level System Design and Analysis</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>D. P.</given-names>
            <surname>Attard</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          .
          <article-title>A Monitoring Tool for a Branching-Time Logic</article-title>
          . In Y. Falcone and C. Sánchez, editors,
          <source>Proc. of the 16th International Conference on Runtime Verification (RV)</source>
          , volume
          <volume>10012</volume>
          <source>of LNCS</source>
          , pages
          <fpage>473</fpage>
          -
          <lpage>481</lpage>
          . Springer,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>D. P.</given-names>
            <surname>Attard</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          .
          <article-title>Trace partitioning and local monitoring for asynchronous components</article-title>
          . In A. Cimatti and M. Sirjani, editors,
          <source>Software Engineering and Formal Methods - 15th International Conference, SEFM</source>
          <year>2017</year>
          , Trento, Italy, September 4-
          <issue>8</issue>
          ,
          <year>2017</year>
          , Proceedings, volume
          <volume>10469</volume>
          <source>of LNCS</source>
          , pages
          <fpage>219</fpage>
          -
          <lpage>235</lpage>
          . Springer,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>H.</given-names>
            <surname>Barringer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Falcone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Havelund</surname>
          </string-name>
          , G. Reger, and
          <string-name>
            <given-names>D. E.</given-names>
            <surname>Rydeheard</surname>
          </string-name>
          .
          <article-title>Quantified event automata: Towards expressive and efficient runtime monitors</article-title>
          . In D. Giannakopoulou and D. Méry, editors,
          <source>FM</source>
          , volume
          <volume>7436</volume>
          of Lecture Notes in Computer Science, pages
          <fpage>68</fpage>
          -
          <lpage>84</lpage>
          . Springer,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>H.</given-names>
            <surname>Barringer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Goldberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Havelund</surname>
          </string-name>
          , and
          <string-name>
            <given-names>K.</given-names>
            <surname>Sen</surname>
          </string-name>
          .
          <article-title>Rule-based runtime verification</article-title>
          . In B. Steffen and G. Levi, editors,
          <source>Verification, Model Checking, and Abstract Interpretation</source>
          , volume
          <volume>2937</volume>
          <source>of LNCS</source>
          , pages
          <fpage>44</fpage>
          -
          <lpage>57</lpage>
          . Springer Berlin Heidelberg,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>E.</given-names>
            <surname>Bartocci</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Falcone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          , and
          <string-name>
            <given-names>G.</given-names>
            <surname>Reger</surname>
          </string-name>
          .
          <article-title>Introduction to runtime verification</article-title>
          . In E. Bartocci and Y. Falcone, editors,
          <source>Lectures on Runtime Verification - Introductory and Advanced Topics</source>
          , volume
          <volume>10457</volume>
          <source>of LNCS</source>
          , pages
          <fpage>1</fpage>
          -
          <lpage>33</lpage>
          . Springer,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>G.</given-names>
            <surname>Boudol</surname>
          </string-name>
          and
          <string-name>
            <given-names>K. G.</given-names>
            <surname>Larsen</surname>
          </string-name>
          .
          <article-title>Graphical versus logical specifications</article-title>
          .
          <source>Theor. Comput. Sci.</source>
          ,
          <volume>106</volume>
          (
          <issue>1</issue>
          ):
          <fpage>3</fpage>
          -
          <lpage>20</lpage>
          ,
          <year>1992</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>J.</given-names>
            <surname>Bradfield</surname>
          </string-name>
          and
          <string-name>
            <given-names>C.</given-names>
            <surname>Stirling</surname>
          </string-name>
          . Chapter 4
          <article-title>- modal logics and mu-calculi: An introduction</article-title>
          . In J.
          <string-name>
            <surname>Bergstra</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Ponse</surname>
          </string-name>
          , and S. Smolka, editors,
          <source>Handbook of Process Algebra</source>
          , pages
          <fpage>293</fpage>
          -
          <lpage>330</lpage>
          . Elsevier Science, Amsterdam,
          <year>2001</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>G.</given-names>
            <surname>Brat</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Drusinsky</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Giannakopoulou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Goldberg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Havelund</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Lowry</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Pasareanu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Venet</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Visser</surname>
          </string-name>
          , and
          <string-name>
            <given-names>R.</given-names>
            <surname>Washington</surname>
          </string-name>
          .
          <article-title>Experimental evaluation of verification and validation tools on Martian rover software</article-title>
          .
          <source>FMSD</source>
          ,
          <volume>25</volume>
          (
          <issue>2-3</issue>
          ):
          <fpage>167</fpage>
          -
          <lpage>198</lpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>E.</given-names>
            <surname>Chang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Manna</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Pnueli</surname>
          </string-name>
          .
          <article-title>Characterization of temporal property classes</article-title>
          .
          <source>In ALP</source>
          , volume
          <volume>623</volume>
          <source>of LNCS</source>
          , pages
          <fpage>474</fpage>
          -
          <lpage>486</lpage>
          . Springer-Verlag,
          <year>1992</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>C.</given-names>
            <surname>Cini</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          .
          <article-title>An LTL proof system for runtime verification</article-title>
          . In C. Baier and C. Tinelli, editors,
          <source>Proc. of the 21st International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS)</source>
          , volume
          <volume>9035</volume>
          <source>of LNCS</source>
          , pages
          <fpage>581</fpage>
          -
          <lpage>595</lpage>
          . Springer,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>E. M.</given-names>
            <surname>Clarke</surname>
          </string-name>
          ,
          <string-name>
            <surname>Jr.</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          <string-name>
            <surname>Grumberg</surname>
            , and
            <given-names>D. A.</given-names>
          </string-name>
          <string-name>
            <surname>Peled</surname>
          </string-name>
          . Model Checking. MIT Press,
          <year>1999</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <surname>B. D'Angelo</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Sankaranarayanan</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Sánchez</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          <string-name>
            <surname>Robinson</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Finkbeiner</surname>
            ,
            <given-names>H. B.</given-names>
          </string-name>
          <string-name>
            <surname>Sipma</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Mehrotra</surname>
            , and
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Manna</surname>
          </string-name>
          . Lola:
          <article-title>Runtime monitoring of synchronous systems</article-title>
          .
          <source>In TIME</source>
          , pages
          <fpage>166</fpage>
          -
          <lpage>174</lpage>
          . IEEE,
          <year>June 2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>D. de Frutos-Escrig</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Gregorio-Rodríguez</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Palomino</surname>
            , and
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Romero-Hernández</surname>
          </string-name>
          .
          <article-title>Unifying the linear time-branching time spectrum of process semantics</article-title>
          .
          <source>Logical Methods in Computer Science</source>
          ,
          <volume>9</volume>
          (
          <issue>2</issue>
          ),
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>N.</given-names>
            <surname>Decker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Leucker</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>Thoma</surname>
          </string-name>
          . jUnitRV - Adding Runtime Verification to jUnit.
          <source>In NASA Formal Methods</source>
          , volume
          <volume>7871</volume>
          <source>of LNCS</source>
          , pages
          <fpage>459</fpage>
          -
          <lpage>464</lpage>
          . Springer,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>A.</given-names>
            <surname>Desai</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Dreossi</surname>
          </string-name>
          , and
          <string-name>
            <given-names>S. A.</given-names>
            <surname>Seshia</surname>
          </string-name>
          .
          <article-title>Combining model checking and runtime verification for safe robotics</article-title>
          . In S. Lahiri and G. Reger, editors,
          <source>Runtime Verification</source>
          , pages
          <fpage>172</fpage>
          -
          <lpage>189</lpage>
          , Cham,
          <year>2017</year>
          . Springer International Publishing.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          .
          <article-title>A Theory of Monitors (Extended Abstract)</article-title>
          .
          <source>In Foundations of Software Science and Computation Structures - 19th International Conference</source>
          , FOSSACS, Eindhoven, The Netherlands, volume
          <volume>9634</volume>
          <source>of LNCS</source>
          , pages
          <fpage>145</fpage>
          -
          <lpage>161</lpage>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          .
          <article-title>Consistently-detecting monitors</article-title>
          .
          <source>In 28th International Conference on Concurrency Theory (CONCUR)</source>
          , volume
          <volume>85</volume>
          <source>of LIPIcs</source>
          , pages
          <volume>8</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>8</lpage>
          :
          <fpage>19</fpage>
          .
          <string-name>
            <surname>Schloss</surname>
            <given-names>Dagstuhl</given-names>
          </string-name>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Achilleos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. P.</given-names>
            <surname>Attard</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I.</given-names>
            <surname>Cassar</surname>
          </string-name>
          ,
          <string-name>
            <surname>D.</surname>
          </string-name>
          <article-title>Della Monica, and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          .
          <article-title>A Foundation for Runtime Monitoring</article-title>
          .
          <source>In Proc. of the 17th International Conference on Runtime Verification (RV)</source>
          , volume
          <volume>10548</volume>
          <source>of LNCS</source>
          , pages
          <fpage>8</fpage>
          -
          <lpage>29</lpage>
          . Springer,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          .
          <article-title>On verifying Hennessy-Milner logic with recursion at runtime</article-title>
          . In E. Bartocci and R. Majumdar, editors,
          <source>Runtime Verification - 6th International Conference</source>
          , RV 2015 Vienna, Austria,
          <source>September 22-25</source>
          ,
          <year>2015</year>
          . Proceedings, volume
          <volume>9333</volume>
          <source>of LNCS</source>
          , pages
          <fpage>71</fpage>
          -
          <lpage>86</lpage>
          . Springer,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Aceto</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Ingólfsdóttir</surname>
          </string-name>
          .
          <article-title>Monitorability for the Hennessy-Milner logic with recursion</article-title>
          .
          <source>Formal Methods in System Design</source>
          ,
          <volume>51</volume>
          (
          <issue>1</issue>
          ):
          <fpage>87</fpage>
          -
          <lpage>116</lpage>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>A.</given-names>
            <surname>Francalanza</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Seychell</surname>
          </string-name>
          .
          <source>Synthesising Correct Concurrent Runtime Monitors. Formal Methods in System Design</source>
          , pages
          <fpage>1</fpage>
          -
          <lpage>36</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>K.</given-names>
            <surname>Kejstová</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Ročkai</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Barnat</surname>
          </string-name>
          .
          <article-title>From model checking to runtime verification and back</article-title>
          . In S. Lahiri and G. Reger, editors,
          <source>Runtime Verification, LNCS</source>
          , pages
          <fpage>225</fpage>
          -
          <lpage>240</lpage>
          , Cham,
          <year>2017</year>
          . Springer International Publishing.
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>M.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Viswanathan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kannan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I.</given-names>
            <surname>Lee</surname>
          </string-name>
          , and
          <string-name>
            <given-names>O.</given-names>
            <surname>Sokolsky</surname>
          </string-name>
          .
          <article-title>Java-MaC: A Run-Time Assurance Approach for Java Programs</article-title>
          . FMSD,
          <volume>24</volume>
          (
          <issue>2</issue>
          ):
          <fpage>129</fpage>
          -
          <lpage>155</lpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [30]
          <string-name>
            <given-names>M.</given-names>
            <surname>Leucker</surname>
          </string-name>
          and
          <string-name>
            <given-names>C.</given-names>
            <surname>Schallhart</surname>
          </string-name>
          .
          <article-title>A brief account of Runtime Verification</article-title>
          . JLAP,
          <volume>78</volume>
          (
          <issue>5</issue>
          ):
          <fpage>293</fpage>
          -
          <lpage>303</lpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>Z.</given-names>
            <surname>Manna</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Pnueli</surname>
          </string-name>
          .
          <article-title>Completing the Temporal Picture</article-title>
          . TCS,
          <volume>83</volume>
          (
          <issue>1</issue>
          ):
          <fpage>97</fpage>
          -
          <lpage>130</lpage>
          ,
          <year>1991</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          [32]
          <string-name>
            <given-names>P. O.</given-names>
            <surname>Meredith</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Jin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Griffith</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Chen</surname>
          </string-name>
          , and
          <string-name>
            <surname>G. Roşu.</surname>
          </string-name>
          <article-title>An overview of the MOP runtime verification framework</article-title>
          .
          <source>STTT</source>
          ,
          <volume>14</volume>
          (
          <issue>3</issue>
          ):
          <fpage>249</fpage>
          -
          <lpage>289</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          [33]
          <string-name>
            <given-names>A.</given-names>
            <surname>Pnueli</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Zaks</surname>
          </string-name>
          .
          <article-title>PSL model checking and run-time verification via testers</article-title>
          . In J. Misra,
          <string-name>
            <given-names>T.</given-names>
            <surname>Nipkow</surname>
          </string-name>
          , and E. Sekerinski, editors,
          <source>FM 2006: Formal Methods, 14th International Symposium on Formal Methods</source>
          , volume
          <volume>4085</volume>
          <source>of LNCS</source>
          , pages
          <fpage>573</fpage>
          -
          <lpage>586</lpage>
          . Springer,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          [34]
          <string-name>
            <given-names>G.</given-names>
            <surname>Reger</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H. C.</given-names>
            <surname>Cruz</surname>
          </string-name>
          , and
          <string-name>
            <surname>D. E. Rydeheard.</surname>
          </string-name>
          <article-title>MarQ: Monitoring at runtime with QEA</article-title>
          . In C. Baier and C. Tinelli, editors,
          <source>Proc. of the 21st International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS)</source>
          , volume
          <volume>9035</volume>
          <source>of LNCS</source>
          , pages
          <fpage>596</fpage>
          -
          <lpage>610</lpage>
          . Springer,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          [35]
          <string-name>
            <surname>R. J. van Glabbeek.</surname>
          </string-name>
          <article-title>The linear time-branching time spectrum II</article-title>
          .
          <source>In Proc. of the 4th International Conference on Concurrency Theory (CONCUR)</source>
          , volume
          <volume>715</volume>
          of Lecture Notes in Computer Science, pages
          <fpage>66</fpage>
          -
          <lpage>81</lpage>
          . Springer,
          <year>1993</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          [36]
          <string-name>
            <surname>R. J. van Glabbeek.</surname>
          </string-name>
          <article-title>The linear time-branching time spectrum I. In Handbook of Process Algebra</article-title>
          , pages
          <fpage>3</fpage>
          -
          <lpage>99</lpage>
          . North-Holland/Elsevier,
          <year>2001</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          [37]
          <string-name>
            <given-names>S.</given-names>
            <surname>Varvaressos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Vaillancourt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Gaboury</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. Blondin</given-names>
            <surname>Massé</surname>
          </string-name>
          , and
          <string-name>
            <given-names>S.</given-names>
            <surname>Hallé</surname>
          </string-name>
          .
          <article-title>Runtime monitoring of temporal logic properties in a platform game</article-title>
          .
          <source>In Runtime Verification</source>
          , volume
          <volume>8174</volume>
          <source>of LNCS</source>
          , pages
          <fpage>346</fpage>
          -
          <lpage>351</lpage>
          . Springer,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>