<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Are Formal Contracts a useful Digital Twin of Software Systems?</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Jonas Schifl</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Alexander Weigl</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Karlsruhe Institute of Technology</institution>
          ,
          <addr-line>Karlsruhe</addr-line>
          ,
          <country country="DE">Germany</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Digital Twins are a trend topic in the industry today to either manage runtime information or forecast properties of devices and products. The techniques for Digitial Twins are already employed in several disciplines of formal methods, in particular, formal verification, runtime verification and specification inference. In this paper, we connect the Digital Twin concept and existing research areas in the field of formal methods. We sketch how digital twins for software-centric systems can be forged from existing formal methods.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Formal Verification</kwd>
        <kwd>Runtime Verification</kwd>
        <kwd>Specification Mining</kwd>
        <kwd>Temporal Logics</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>System
(Physical Domain)</p>
      <p>Digital Twin
Generic Model</p>
      <p>Configuration</p>
      <p>Update</p>
      <p>Prediction
Contribution In this paper, we apply several formal methods and tools to establish a
framework for Digital Twins of reactive systems using formal software contracts. Therefore,
we focus on twinning the software behavior of a reactive system operation according to
its formal model, which we define through contracts.</p>
      <p>The Digital Twin allows forecasting properties that depend on the behavior of the
system. However, forecasting is not limited to the specific instance to which it is twinned.
Moreover, knowledge learned from observation can be shared between Digital Twins, and
What-if analyses are available. To achieve this, we show how existing formal methods,
in particular formal verification, runtime verification, specification mining and program
repair, can be combined for a Digital Twin.</p>
      <p>A meta-model for Digital Twin For this paper, we assume a simple meta-model of
Digital Twins, sketched in Figure 1. This meta-model consists of the original system that
exists in the real world. Such systems can be cyber-physical systems, like cars, automated
production systems, or energy systems. For the purposes of this discussion, we only
consider pure software systems.</p>
      <p>
        The Digital Twin is then a digital representation of an instance of a system. In case of a
cyber-physical system, this can be a specific car or an individual transformer in the energy
system. Coming from the domain of formal methods, we build the Digital Twin by using
symbolic logic based models. We will later use contract-languages to represent the system.
For software systems, the Digital Twin can be split into a generic part that is valid for
all instances of the system parameters, and instance-specific configuration. For example,
the generic consists of the classes and their contracts, but their use and composition is
determined by the instance’s configuration. Note that the Digital Twin also contains
information about the environment. This is similar to the distinction between ABox and
TBox in description logics. The TBox, the generic part, holds the axioms that are valid
for all instances. The ABox, on the other hand, contains the information for the specific
instance [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. The TBox determines which parts of the ABox are relevant.
      </p>
      <sec id="sec-1-1">
        <title>Operations</title>
        <p>tions:</p>
        <p>The representation of a system as a Digital Twin should enable two
opera• Prediction of selected properties. Prediction means reasoning whether a property
holds for a specific instance or, due to the symbolic representation, for a family
of instances. Moreover, we can perform “What-If” analyses: assuming a diferent
configuration, we can predict properties on diferent instances, after change on
the system. We can also model a transfer to a diferent environment. Prediction
can lead to efects on the system, e.g., only configurations with a good predicted
performance are deployed.
• The Update process consists of collecting and aggregating information from the
(real-world) instance and transferring these into the configuration part of the model.
Desired Properties A Digital Twin also has to fulfill some properties. First, it should
be faithful in the sense that the derived predictions are precise. If precision is too hard
to achieve, we claim at least soundness: The prediction should be a pessimistic (or
conservative) evaluation with respect to the safety analysis. The reasoning based on
the Digital Twin should never attest safety falsely; if in doubt, it should rather predict
non-existing safety issues.</p>
        <p>Additionally, compositionality of a Digital Twin brings the same advantages as for the
system. Systems can be built up by composition of multiple systems. The same is desirable
for the Digital Twins: the Digital Twin of the top system is, ideally, a composition of the
Digital Twins of the sub systems.</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>2. Contracts for Reactive Systems</title>
      <p>In the following, we instantiate the Digital Twin meta-model for reactive software systems.
For this, we define the individual components: the system and the formal model as well
as the prediction and update processes.</p>
      <sec id="sec-2-1">
        <title>2.1. The system.</title>
        <p>
          A reactive system is characterized by the following properties (cf. [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ]): Their runtime
is infinite or undetermined, and they interact with the physical world, e.g., by reading
sensor values and controlling actuators. Typical representatives for reactive systems are
embedded controllers in the automation or automotive domain.
        </p>
        <p>
          Reactive systems can be constructed via composition from smaller systems. For example,
Lingua Franca [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] defines reactive systems, named reactors, that can be built of other
reactors, or an executable program fragment. A similar terminology can be found in
IEC 61131-3 (the standard for programming languages for automated production systems)
as Function Block Diagrams. In this paper, we assume the system model of [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ], in which
every system has an interface (input and output variables) and is composed of connected
subsystems.
        </p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. The Digital Twin.</title>
        <p>
          Cimatti et al. [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ] provide OCRA, a tool that enables a design-by-contract methodology
for reactive systems by defining components, their composition and associated contracts.
OCRA focuses on the refinement relationship, thus verification of the program code
against contract is left out.
        </p>
        <p>Using the OCRA model, the Digital Twin of a system is its contract, which consists of
two properties given in Linear Temporal Logic (LTL). The first property is an assumption
on the system inputs. The second one is a guarantee of the system outputs under the
given assumptions. Note that LTL is just the language we use for describing sets of traces.
It can be replaced by any logic on traces (see below). The advantage of LTL is acceptance
by a wide range of tools.</p>
        <p>In addition to the contract, the Digital Twin is also aware of the structural architecture
of the software given by the composition of subsystems and the information flow between
them. Parameterization (or configuration) of a system is established via its input variables
and by the selection of the sub-system during the composition.</p>
        <p>The contract is a syntactical notion that describe the set of allowed behaviors of a
system under a specified set of input traces. The behavior of a correct system is a subset of
the allowed behavior. Often, due to the determinism in the system and the indeterminism
(or abstraction) in the contract, the allowed behavior is indeed a strict superset of the
actual actual behavior. Additionally, a system composition gives a refinement obligation,
in which the composition, i.e., the collective of sub-systems, needs to conjointly fulfill the
super system’s contracts. Furthermore, the systems within a composition also need to
be compatible with each other. Due to the over-approximating character of contracts,
the Digital Twin tends to be imprecise, but sound for safety analyses. However, it is
not possible to predict liveness properties. Note that proving the correctness of the
system is sometimes infeasible, e.g., due to unavailability of the source code, complexity
of the system (floating point, state explosion) or lack of knowledge of the environment.
Additionally, we do not assume that the system is always compliant with the contract,
especially, when the contracts are automatically mined from observation, or extracted
from natural language and not carefully designed by the system creator and operators.
This increases the need for a robust update that incrementally tighten the coupling
between the instance of the system and Digital Twin built from contracts.</p>
      </sec>
      <sec id="sec-2-3">
        <title>2.3. Update</title>
        <p>There are several update operations. The most simple one is detecting the parameterization
of the instance, i.e., determining the values of the diferent inputs.</p>
        <p>
          Moreover, the assumptions and guarantees in the contracts at the top-level and
subsystems needs to be validated during operation to ensure that the system operates in
its expected boundaries. In particular, we check whether a system is correctly invoked
by its outer context or environment, and whether it behaves as specified. To achieve
this, Runtime Verification, such as [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ] for LTL, can be used. However, from Runtime
Verification we can only learn about violation of or adherence to a property.
        </p>
        <p>
          Normally, we want to establish the correctness of a system statically, but some properties
are too hard to be established in advance. In the best case, unexpected
specificationviolating runtime traces are recorded to enable specification mining. Specification mining
is the discipline to extract LTL properties from traces. For example, Tuxedo [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ] is a
pattern-based approach, that instantiates LTL patterns with state formulas and validates
them against the traces. An instantiated pattern must be fulfilled by a specified ratio
of the observed traces to be considered valid. The valid instantiated patterns help to
adjust the contracts. For example, in the case of a contract violation, we might consider
weakening the assumptions of a system. On the other hand, specification mining may also
be applied where the contracts are fulfilled. In this case, it can lead to a tightening of
the contracts for a specific configuration or environment. Besides of mining specifications
from recorded traces, there are tools for extracting LTL properties from natural languages,
e.g. requirement documentation or user commands [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ].
        </p>
      </sec>
      <sec id="sec-2-4">
        <title>2.4. Prediction</title>
        <p>The prediction operation of our Digital Twin can be reduced to the well-established
model-checking problem which allows us to verify the validity of a (LTL) property in a
given Kripke structure. This covers the validation that each contract is adhered to by the
associated system, and that systems are composed in a valid fashion.</p>
        <p>But we benefit from the information learned during the Update operation: We can
testify the contract adherence specifically in the used system configuration, and thus, we
can save verification run-time without sufering a loss of verification validity for a specific
instance. Furthermore, it may be beneficial to limit the verification process only to the
recorded and mined environment of the instance to be validated.</p>
        <p>Of course, the Digital Twin allows to conduct “What-If” analyses by altering the
parameter of a system, or implementing it into a diferent environment. The latter simply
requires changing the mined properties. This also requires tracking the origin of the
formulas within the contracts.</p>
        <p>
          But we are not limited to model-checking. Testing or simulation are also in reach by
using the contracts. In particular, formal contracts enable testing and simulation even for
software which is not executable in a simulation environment (e.g., due to inaccessible
resources). There are also more sophisticated techniques. For example, Program Repair
considers altering programs such that the program fulfills its contract. For this, the source
code of the program is mutated until a suitable candidate is found [
          <xref ref-type="bibr" rid="ref10 ref9">9, 10</xref>
          ]. A similar
discipline is Parameter Synthesis [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ].
        </p>
        <p>Limitations By using assume-guarantee contracts and LTL as the specification language,
we focus on describing behavior on a certain abstraction level. This allows us to reason
well on the possibility to reach bad system states, but other properties are not predictable.
For example, security properties that are expressible as reachability are covered, i.e.,
integrity of the system, but confidentiality (e.g., secure information flow), and availability
(liveness properties) are not. Moreover, runtime predictions, e.g., worst case execution
time (WCET), are also not in our setup, due to the abstraction of time in LTL.</p>
        <sec id="sec-2-4-1">
          <title>2.4.1. Other contract languages.</title>
          <p>
            Note that OCRA uses LTL, but its approach is not limited to a particular contract
grammar. There are many variants of temporal logics. For example, Metric Temporal
Logic (MTL) extends LTL to include real-time capabilities [
            <xref ref-type="bibr" rid="ref12">12</xref>
            ]. In MTL, temporal
operators have an additional time span in which the formula needs to be fulfilled. MTL
formulas are evaluated on “data words”—event traces with an explicit time value. MTL is
well studied for runtime verification [
            <xref ref-type="bibr" rid="ref13 ref14">13, 14</xref>
            ] and specification mining [
            <xref ref-type="bibr" rid="ref15">15</xref>
            ]. Additionally,
Runtime Verification for MTL can also be quantitative by measuring how large the
violation of the specification is [
            <xref ref-type="bibr" rid="ref16">16</xref>
            ].
          </p>
          <p>
            Temporal logic formulas tend to be very hard to understand. Therefore, an additional
goal is comprehensibility of the contracts and thus the Digital Twin. For example,
Generalized Test Tables (GTTs) [
            <xref ref-type="bibr" rid="ref17">17</xref>
            ] are an engineer-friendly specification language
derived from test tables used in the automation industry. A GTT describes a behavior in
a particular scenario without the aspiration to be a complete specification. The rows in
a GTT are the sequential steps of a test protocol, where each step consists of multiple
assumptions and assertions (the table columns) for each input-output variable. Due
to the table-based format, a fine-grained runtime monitoring is possible [
            <xref ref-type="bibr" rid="ref18">18</xref>
            ] in which
the violation of single constraints can be tracked. Specification mining is currently not
available, but might be adopted from approaches for learning finite-state machine based
specifications (e.g., [
            <xref ref-type="bibr" rid="ref19">19</xref>
            ]).
          </p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Closing Remarks</title>
      <p>
        In this paper, we present the idea of using formal contracts – existing in current
designby-contract approaches – for the representation of Digital Twins. Due to tool support, we
focus on reactive systems and LTL, but the principle is also applicable to batch systems.
For example, the Java Modeling Language (JML) [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] is an established specification
language for Java source code with support of deductive verification [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ] and runtime
verification [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ]. Specification mining (of method contracts) based on runtime information
is currently not well-researched.
      </p>
      <p>
        Software and its refinement. When we fade out the physical environment and
concentrate on the software, we can state that the software, as it is digital, is itself the most
precise definition of its behavior, and can also been seen as its own contract. Every other
contract for a system over-approximates. But the abstraction of contracts is needed,
as it allows us to hide the complexity of the implementation. Indeed, more abstract
software models are often used in software engineering: For example, the buildability of
a user-configurable product is defined by a feature model. Feature models reflect the
compatibility of software modules, which also depends on the behavior of the software.
Evolution of Digital Twins with Relational Verification. For our framework, we have
only considered the classical specification and verification of single traces. A possible
extension is the verification of relational properties [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] (or multi-properties). A relational
property describes the relationship between multiple runs of the same or diferent systems.
For example, regression verification – a relaxed program equivalence – is a relational
property which helps to identify the introduction of unintended behavior during the
co-evolution of hard- and software. Regression verification requires a description of
the relationship between the common behavior of both systems [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ]. As the regression
contracts help with the co-evolution by coupling the old to the new version, they also
support the evolution step of the Digital Twins by coupling mined knowledge from the
old twin to the new twin.
      </p>
    </sec>
    <sec id="sec-4">
      <title>Acknowledgments</title>
      <p>This work was supported by funding of the Helmholtz Association (HGF) through the
Competence Center for Applied Security Technology (KASTEL) and by the research
project SofDCar (19S21002), which is funded by the German Federal Ministry for Economic
Afairs and Climate Action</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>A.</given-names>
            <surname>Löcklin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Müller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Jung</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Jazdi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>White</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Weyrich</surname>
          </string-name>
          ,
          <article-title>Digital twin for verification and validation of industrial automation systems - a survey</article-title>
          ,
          <source>in: 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)</source>
          , volume
          <volume>1</volume>
          ,
          <year>2020</year>
          , pp.
          <fpage>851</fpage>
          -
          <lpage>858</lpage>
          . doi:
          <volume>10</volume>
          .1109/ETFA46521.
          <year>2020</year>
          .
          <volume>9212051</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>G. D.</given-names>
            <surname>Giacomo</surname>
          </string-name>
          ,
          <string-name>
            <surname>M.</surname>
          </string-name>
          <article-title>Lenzerini, TBox and ABox reasoning in expressive description logics</article-title>
          , in: L.
          <string-name>
            <surname>Padgham</surname>
            , E. Franconi,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Gehrke</surname>
            ,
            <given-names>D. L.</given-names>
          </string-name>
          <string-name>
            <surname>McGuinness</surname>
            ,
            <given-names>P. F.</given-names>
          </string-name>
          PatelSchneider (Eds.),
          <source>Proceedings of the 1996 International Workshop on Description Logics, November 2-4</source>
          ,
          <year>1996</year>
          , Cambridge, MA, USA, volume WS-
          <volume>96</volume>
          -05
          <source>of AAAI Technical Report</source>
          , AAAI Press,
          <year>1996</year>
          , pp.
          <fpage>37</fpage>
          -
          <lpage>48</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>N.</given-names>
            <surname>Halbwachs</surname>
          </string-name>
          ,
          <article-title>Synchronous programming of reactive systems</article-title>
          , in: A.
          <string-name>
            <surname>J. Hu</surname>
          </string-name>
          , M. Y. Vardi (Eds.), Computer Aided Verification, 10th International Conference, CAV '
          <fpage>98</fpage>
          , Vancouver, BC, Canada, June 28 - July 2,
          <year>1998</year>
          , Proceedings, volume
          <volume>1427</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>1998</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>16</lpage>
          . doi:
          <volume>10</volume>
          .1007/BFb0028726.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>M.</given-names>
            <surname>Lohstroh</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Menard</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bateni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. A.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <article-title>Toward a Lingua Franca for deterministic concurrent systems</article-title>
          ,
          <source>ACM Trans. Embed. Comput. Syst</source>
          .
          <volume>20</volume>
          (
          <year>2021</year>
          ). doi:
          <volume>10</volume>
          .1145/3448128.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>A.</given-names>
            <surname>Cimatti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Dorigatti</surname>
          </string-name>
          , S. Tonetta,
          <article-title>OCRA: A tool for checking the refinement of temporal contracts</article-title>
          , in: E. Denney,
          <string-name>
            <given-names>T.</given-names>
            <surname>Bultan</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . Zeller (Eds.),
          <year>2013</year>
          28th IEEE/ACM International Conference on
          <source>Automated Software Engineering, ASE</source>
          <year>2013</year>
          , Silicon Valley, CA, USA, November
          <volume>11</volume>
          -
          <issue>15</issue>
          ,
          <year>2013</year>
          , IEEE,
          <year>2013</year>
          , pp.
          <fpage>702</fpage>
          -
          <lpage>705</lpage>
          . doi:
          <volume>10</volume>
          .1109/ASE.
          <year>2013</year>
          .
          <volume>6693137</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bauer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Leucker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Schallhart</surname>
          </string-name>
          ,
          <article-title>Runtime verification for LTL and TLTL</article-title>
          ,
          <source>ACM Trans. Softw. Eng. Methodol</source>
          .
          <volume>20</volume>
          (
          <year>2011</year>
          ). doi:
          <volume>10</volume>
          .1145/2000799.2000800.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>C.</given-names>
            <surname>Lemieux</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Park</surname>
          </string-name>
          , I. Beschastnikh,
          <article-title>General LTL specification mining (T)</article-title>
          , in: M. B.
          <string-name>
            <surname>Cohen</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Grunske</surname>
          </string-name>
          , M. Whalen (Eds.),
          <source>30th IEEE/ACM International Conference on Automated Software Engineering, ASE</source>
          <year>2015</year>
          ,
          <article-title>Lincoln</article-title>
          ,
          <string-name>
            <surname>NE</surname>
          </string-name>
          , USA, November 9-
          <issue>13</issue>
          ,
          <year>2015</year>
          , IEEE Computer Society,
          <year>2015</year>
          , pp.
          <fpage>81</fpage>
          -
          <lpage>92</lpage>
          . doi:
          <volume>10</volume>
          .1109/ASE.
          <year>2015</year>
          .
          <volume>71</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>C.</given-names>
            <surname>Wang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Ross</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Kuo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Katz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Barbu</surname>
          </string-name>
          ,
          <article-title>Learning a natural-language to LTL executable semantic parser for grounded robotics</article-title>
          , in: J.
          <string-name>
            <surname>Kober</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          <string-name>
            <surname>Ramos</surname>
            ,
            <given-names>C. J.</given-names>
          </string-name>
          <string-name>
            <surname>Tomlin</surname>
          </string-name>
          (Eds.), 4th Conference on Robot Learning,
          <source>CoRL</source>
          <year>2020</year>
          ,
          <fpage>16</fpage>
          -
          <lpage>18</lpage>
          November 2020, Virtual Event / Cambridge, MA, USA, volume
          <volume>155</volume>
          <source>of Proceedings of Machine Learning Research, PMLR</source>
          ,
          <year>2020</year>
          , pp.
          <fpage>1706</fpage>
          -
          <lpage>1718</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>M.</given-names>
            <surname>Brizzio</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Degiovanni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Cordy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Papadakis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Aguirre</surname>
          </string-name>
          ,
          <article-title>Automated repair of unrealisable LTL specifications guided by model counting</article-title>
          ,
          <source>CoRR abs/2105</source>
          .12595 (
          <year>2021</year>
          ). arXiv:
          <volume>2105</volume>
          .
          <fpage>12595</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>V.</given-names>
            <surname>Mironovich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Buzdalov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Vyatkin</surname>
          </string-name>
          ,
          <article-title>Automatic generation of function block applications using evolutionary algorithms: Initial explorations</article-title>
          ,
          <source>in: 15th IEEE International Conference on Industrial Informatics, INDIN</source>
          <year>2017</year>
          , Emden, Germany,
          <source>July 24-26</source>
          ,
          <year>2017</year>
          , IEEE,
          <year>2017</year>
          , pp.
          <fpage>700</fpage>
          -
          <lpage>705</lpage>
          . doi:
          <volume>10</volume>
          .1109/INDIN.
          <year>2017</year>
          .
          <volume>8104858</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>P.</given-names>
            <surname>Bezdek</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Benes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Barnat</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Cerná</surname>
          </string-name>
          ,
          <article-title>LTL parameter synthesis of parametric timed automata</article-title>
          , in: R. D.
          <string-name>
            <surname>Nicola</surname>
          </string-name>
          , E. Kühn (Eds.),
          <source>Software Engineering and Formal Methods - 14th International Conference, SEFM</source>
          <year>2016</year>
          ,
          <article-title>Held as Part of STAF 2016</article-title>
          , Vienna, Austria,
          <source>July 4-8</source>
          ,
          <year>2016</year>
          , Proceedings, volume
          <volume>9763</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2016</year>
          , pp.
          <fpage>172</fpage>
          -
          <lpage>187</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>319</fpage>
          -41591-8\ _
          <fpage>12</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>R.</given-names>
            <surname>Koymans</surname>
          </string-name>
          ,
          <article-title>Specifying real-time properties with metric temporal logic</article-title>
          ,
          <source>Real Time Syst</source>
          .
          <volume>2</volume>
          (
          <year>1990</year>
          )
          <fpage>255</fpage>
          -
          <lpage>299</lpage>
          . doi:
          <volume>10</volume>
          .1007/BF01995674.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>P.</given-names>
            <surname>Thati</surname>
          </string-name>
          , G. Rosu,
          <article-title>Monitoring algorithms for metric temporal logic specifications</article-title>
          , in: K. Havelund, G. Rosu (Eds.),
          <source>Proceedings of the Fourth Workshop on Runtime Verification, RV@ETAPS</source>
          <year>2004</year>
          , Barcelona, Spain, April 3,
          <year>2004</year>
          , volume
          <volume>113</volume>
          of Electronic Notes in Theoretical Computer Science, Elsevier,
          <year>2004</year>
          , pp.
          <fpage>145</fpage>
          -
          <lpage>162</lpage>
          . doi:
          <volume>10</volume>
          .1016/j.entcs.
          <year>2004</year>
          .
          <volume>01</volume>
          .029.
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>H.</given-names>
            <surname>Ho</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Ouaknine</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Worrell</surname>
          </string-name>
          ,
          <article-title>Online monitoring of metric temporal logic</article-title>
          , in: B.
          <string-name>
            <surname>Bonakdarpour</surname>
            ,
            <given-names>S. A.</given-names>
          </string-name>
          <string-name>
            <surname>Smolka</surname>
          </string-name>
          (Eds.),
          <source>Runtime Verification - 5th International Conference, RV 2014</source>
          , Toronto, ON, Canada,
          <source>September 22-25</source>
          ,
          <year>2014</year>
          . Proceedings, volume
          <volume>8734</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2014</year>
          , pp.
          <fpage>178</fpage>
          -
          <lpage>192</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>319</fpage>
          -11164-3\_
          <fpage>15</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>B.</given-names>
            <surname>Hoxha</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Dokhanchi</surname>
          </string-name>
          , G. Fainekos,
          <article-title>Mining parametric temporal logic properties in model-based design for cyber-physical systems</article-title>
          ,
          <source>Int. J. Softw. Tools Technol. Transf</source>
          .
          <volume>20</volume>
          (
          <year>2018</year>
          )
          <fpage>79</fpage>
          -
          <lpage>93</lpage>
          . doi:
          <volume>10</volume>
          .1007/s10009-017-0447-4.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>A.</given-names>
            <surname>Dokhanchi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Hoxha</surname>
          </string-name>
          , G. Fainekos,
          <article-title>On-line monitoring for temporal logic robustness</article-title>
          , in: B.
          <string-name>
            <surname>Bonakdarpour</surname>
            ,
            <given-names>S. A.</given-names>
          </string-name>
          <string-name>
            <surname>Smolka</surname>
          </string-name>
          (Eds.),
          <source>Runtime Verification - 5th International Conference, RV 2014</source>
          , Toronto, ON, Canada,
          <source>September 22-25</source>
          ,
          <year>2014</year>
          . Proceedings, volume
          <volume>8734</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2014</year>
          , pp.
          <fpage>231</fpage>
          -
          <lpage>246</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>319</fpage>
          -11164-3\_
          <fpage>19</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>B.</given-names>
            <surname>Beckert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ulbrich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Vogel-Heuser</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Weigl</surname>
          </string-name>
          ,
          <article-title>Generalized test tables: A domainspecific specification language for automated production systems</article-title>
          , in: H.
          <string-name>
            <surname>Seidl</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          <string-name>
            <surname>Liu</surname>
          </string-name>
          , C. S. Pasareanu (Eds.),
          <source>Theoretical Aspects of Computing - ICTAC</source>
          <year>2022</year>
          - 19th International Colloquium, Tbilisi, Georgia,
          <source>September 27-29</source>
          ,
          <year>2022</year>
          , Proceedings, volume
          <volume>13572</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2022</year>
          , pp.
          <fpage>7</fpage>
          -
          <lpage>13</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>031</fpage>
          -17715-6\_2.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>A.</given-names>
            <surname>Weigl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ulbrich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. S.</given-names>
            <surname>Tyszberowicz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Klamroth</surname>
          </string-name>
          ,
          <article-title>Runtime verification of generalized test tables</article-title>
          , in: A.
          <string-name>
            <surname>Dutle</surname>
          </string-name>
          ,
          <string-name>
            <surname>M. M. Moscato</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Titolo</surname>
            ,
            <given-names>C. A.</given-names>
          </string-name>
          <string-name>
            <surname>Muñoz</surname>
          </string-name>
          , I. Perez (Eds.),
          <source>NASA Formal Methods - 13th International Symposium, NFM</source>
          <year>2021</year>
          ,
          <string-name>
            <given-names>Virtual</given-names>
            <surname>Event</surname>
          </string-name>
          , May
          <volume>24</volume>
          -28,
          <year>2021</year>
          , Proceedings, volume
          <volume>12673</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2021</year>
          , pp.
          <fpage>358</fpage>
          -
          <lpage>374</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>030</fpage>
          -76384-8\_
          <fpage>22</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>T. B. Le</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Lo</surname>
          </string-name>
          ,
          <article-title>Deep specification mining</article-title>
          , in: F.
          <string-name>
            <surname>Tip</surname>
          </string-name>
          , E. Bodden (Eds.),
          <source>Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis</source>
          ,
          <source>ISSTA</source>
          <year>2018</year>
          , Amsterdam, The Netherlands,
          <source>July 16-21</source>
          ,
          <year>2018</year>
          , ACM,
          <year>2018</year>
          , pp.
          <fpage>106</fpage>
          -
          <lpage>117</lpage>
          . doi:
          <volume>10</volume>
          .1145/3213846.3213876.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>G. T.</given-names>
            <surname>Leavens</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Cheon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Clifton</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Ruby</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. R.</given-names>
            <surname>Cok</surname>
          </string-name>
          ,
          <article-title>How the design of JML accommodates both runtime assertion checking and formal verification</article-title>
          ,
          <source>Sci. Comput</source>
          . Program.
          <volume>55</volume>
          (
          <year>2005</year>
          )
          <fpage>185</fpage>
          -
          <lpage>208</lpage>
          . doi:
          <volume>10</volume>
          .1016/j.scico.
          <year>2004</year>
          .
          <volume>05</volume>
          .015.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>W.</given-names>
            <surname>Ahrendt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Beckert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Bubel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Hähnle</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. H.</given-names>
            <surname>Schmitt</surname>
          </string-name>
          , M. Ulbrich (Eds.),
          <source>Deductive Software Verification - The KeY Book - From Theory to Practice</source>
          , volume
          <volume>10001</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2016</year>
          . doi:
          <volume>10</volume>
          .1007/ 978-3-
          <fpage>319</fpage>
          -49812-6.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>F.</given-names>
            <surname>Hussain</surname>
          </string-name>
          , G. T. Leavens,
          <article-title>temporaljmlc: A JML runtime assertion checker extension for specification and checking of temporal properties</article-title>
          , in: J. L.
          <string-name>
            <surname>Fiadeiro</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Gnesi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Maggiolo-Schettini</surname>
          </string-name>
          (Eds.),
          <source>8th IEEE International Conference on Software Engineering and Formal Methods</source>
          ,
          <string-name>
            <surname>SEFM</surname>
          </string-name>
          <year>2010</year>
          , Pisa, Italy,
          <fpage>13</fpage>
          -18
          <source>September</source>
          <year>2010</year>
          , IEEE Computer Society,
          <year>2010</year>
          , pp.
          <fpage>63</fpage>
          -
          <lpage>72</lpage>
          . doi:
          <volume>10</volume>
          .1109/SEFM.
          <year>2010</year>
          .
          <volume>15</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>B.</given-names>
            <surname>Beckert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ulbrich</surname>
          </string-name>
          ,
          <article-title>Trends in relational program verification</article-title>
          , in: P. Müller, I. Schaefer (Eds.),
          <source>Principled Software Development - Essays Dedicated to Arnd Poetzsch-Hefter on the Occasion of his 60th Birthday</source>
          , Springer,
          <year>2018</year>
          , pp.
          <fpage>41</fpage>
          -
          <lpage>58</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>319</fpage>
          -98047-8\_3.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>A.</given-names>
            <surname>Weigl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ulbrich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Cha</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Beckert</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Vogel-Heuser</surname>
          </string-name>
          ,
          <article-title>Relational test tables: A practical specification language for evolution and security</article-title>
          , in: K.
          <string-name>
            <surname>Bae</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          <string-name>
            <surname>Bianculli</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Gnesi</surname>
          </string-name>
          , N. Plat (Eds.),
          <source>FormaliSE@ICSE 2020: 8th International Conference on Formal Methods in Software Engineering</source>
          , Seoul, Republic of Korea,
          <source>July</source>
          <volume>13</volume>
          ,
          <year>2020</year>
          , ACM,
          <year>2020</year>
          , pp.
          <fpage>77</fpage>
          -
          <lpage>86</lpage>
          . doi:
          <volume>10</volume>
          .1145/3372020.3391566.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>