<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Traceability of Information Flow Requirements in Cyber-Physical Systems Engineering</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Christopher Gerking</string-name>
          <email>christopher.gerking@upb.de</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Paderborn University, Heinz Nixdorf Institute Software Engineering Research Group Paderborn</institution>
          ,
          <addr-line>Germany WWW home page: https://</addr-line>
        </aff>
      </contrib-group>
      <abstract>
        <p>The secure information ow between a cyber-physical system and its environment has evolved into a critical factor. To comply with security regulations, engineers of cyber-physical systems need to trace information ow requirements from their speci cation to the software design. However, due to the interdisciplinary engineering of cyber-physical systems and their inherent real-time behavior, requirements speci ed at a discipline-spanning level are hard to verify at the discipline-speci c software design level. In this PhD project, we address this problem based on a speci cation of information ow requirements in model-based systems engineering. We provide a technique to verify if the real-time behavior of software design models ful lls corresponding information ow properties, and trace veri cation results back to the initial requirements. By establishing traceability, we enable engineers to ensure the regulatory compliance of cyber-physical systems. We intend to validate the applicability of our approach in the scope of an Industrial Internet scenario.</p>
      </abstract>
      <kwd-group>
        <kwd>requirements traceability</kwd>
        <kwd>information ow security</kwd>
        <kwd>systems engineering</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>Along with the 4th industrial revolution, cyber-physical production systems are
expected to enable highly dynamic business relationships between companies.
Business partners are no longer pre-de ned, but may change on the y depending
on quality factors like cost, delivery time, or even environmental friendliness. Due
to the dynamic interconnection of systems in the Industrial Internet, security has
evolved into a critical factor because business secrets are at risk of being exposed
to unknown, untrusted business partners. To avoid violations of security policies,
the software that drives a cyber-physical system needs to control the information
ow between the system and its environment. Thus, systems must meet strict
information ow requirements.</p>
      <p>For example, consider a production system that exchanges information with a
cloud-based service market to order materials. In addition, the system has access
to a con dential operating plan that is considered as a business secret. Thus,
it must not be exposed to the outside world. As an example for an information
ow requirement, the production system needs to avoid an illegitimate ow of
con dential information from the operating plan to the public service market.</p>
      <p>
        Engineers of cyber-physical systems need to consider such requirements at
the design stage because mandatory regulations are expected to involve rigorous
security conditions. Before a novel system may start to operate, the compliance
with its regulations needs to be certi ed by authorities [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. This regulatory
compliance often depends on the traceability of requirements, which is de ned
as \the ability to describe and follow the life of a requirement in both a
forwards and backwards direction" [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Traceability enables certi ers to reproduce
that a system complies to its speci ed requirements. Thus, traceability requires
engineers to trace information ow requirements from their speci cation to the
software design.
      </p>
      <p>Model-driven engineering approaches enable software engineers to verify
information ow properties at the level of software design models. Nevertheless,
due to several characteristics of cyber-physical systems, the traceability of
information ow requirements between speci cation and design is hard to establish
by engineers. In the following, we identify three challenges for engineers (C1-C3)
to be addressed in this PhD project.</p>
      <p>
        First of all, beside software engineering, multiple other disciplines are
involved in the development of cyber-physical systems. An integration of these
disciplines at the level of model-based systems engineering [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ] is bene cial for
providing a holistic view on a system under development. However, as shown
in Fig. 1, the integration also introduces an additional, discipline-spanning
conception stage, preceding the software design and implementation stages. During
the system conception, systems engineers create a model that includes an initial
speci cation of the system requirements (cf. Fig. 1). Due to the negative nature
of information ow requirements (describing an illegitimate ow to avoid), a
t-for-purpose speci cation technique is needed.
      </p>
      <p>C1: How can systems engineers specify information
discipline-spanning conception stage?
ow requirements at a
Specification of
Information Flow</p>
      <p>Requirements</p>
      <p>System
Conception</p>
      <p>Software
Design</p>
      <p>Verification of
Information Flow</p>
      <p>Properties</p>
      <p>Implementation</p>
      <p>Legend
Discipline-spanning</p>
      <p>Stage</p>
      <p>Traceability
Discipline-specific</p>
      <p>Stage</p>
      <p>Fig. 1: Stages of the Engineering Process for Cyber-Physical Systems
Furthermore, due to their interaction with the physical environment, the
systems need to satisfy hard real-time constraints. In order to provide sound
results, veri cation techniques for information ow properties need to take this
real-time behavior into account. Any deviation compromises the soundness of
the veri cation and, therefore, prevents reliable requirements traceability.
C2: How can software engineers verify the information ow properties of
software design models under consideration of real-time behavior?</p>
      <p>Finally, from the viewpoint of an individual discipline such as software
engineering, the discipline-spanning conception leads to an increased vagueness
of the requirements speci cation, as it relies on general terms without precise,
discipline-speci c semantics. Vaguely speci ed information ow requirements are
hard to verify, even if software engineers have capable veri cation techniques at
hand. This divergence in terms of abstraction between requirements speci
cation and software design represents an obstacle for the desired traceability of
information ow requirements as depicted in Fig. 1.</p>
      <p>C3: How to establish traceability between the information ow requirements
and the veri cation of information ow properties at the software design
level?
2</p>
    </sec>
    <sec id="sec-2">
      <title>Related Work</title>
      <p>
        Today, the eld of model-based systems engineering is dominated by the
Systems Modeling Language (SysML, [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]). Nejati et al. [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] establish traceability
between safety requirements and SysML models. In contrast, our intention to
verify the information ow security (challenge C2) requires formal semantics of the
underlying modeling language. Therefore, we need to consider software design
models beyond the scope of SysML. Consens [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] is a method for model-based
systems engineering using SysML-compliant models. Consens is capable of
describing information ow between model elements, and supports the transition
from systems engineering to the software design level. Therefore, Consens is a
suitable basis for our needs to consider the information ow of software design
models. As an example, Fig. 2a shows a Consens model of the interactions
between a production system and its environment. Information is owing from the
operating plan to the system, and from the system to the service market.
However, the Consens approach neither supports the speci cation of illegitimate
information ow (challenge C1), as for example between the operating plan and
the service market, nor the traceability to the software design (challenge C3).
      </p>
      <p>
        For the transition from Consens to the software design, Gausemeier et al. [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]
present a derivation of software component models based on
MechatronicUML (MUML, [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]), a model-driven software design method for cyber-physical
systems. As an example, Fig. 2b shows the software design of the production
system in terms of a MUML component model. Components exchange
information over ports. Whereas MUML supports the formal veri cation of safety
      </p>
      <sec id="sec-2-1">
        <title>Service</title>
      </sec>
      <sec id="sec-2-2">
        <title>Market</title>
      </sec>
      <sec id="sec-2-3">
        <title>Production</title>
      </sec>
      <sec id="sec-2-4">
        <title>System</title>
        <p>Legend
Environmental
Element</p>
      </sec>
      <sec id="sec-2-5">
        <title>Operating</title>
      </sec>
      <sec id="sec-2-6">
        <title>Plan</title>
        <p>System</p>
      </sec>
      <sec id="sec-2-7">
        <title>ProductionSystem</title>
      </sec>
      <sec id="sec-2-8">
        <title>Control</title>
      </sec>
      <sec id="sec-2-9">
        <title>Scheduler</title>
        <p>Material</p>
        <p>Flow</p>
        <p>Information</p>
        <p>Flow</p>
        <p>Software
Component</p>
        <p>
          Port
(a) Consens Environment Model
(b) MUML Component Model
properties [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ] under consideration of the real-time behavior of components, it
does not enable the veri cation of information ow properties (challenge C2).
As an example for such a property, the software design depicted in Fig. 2b needs
to ensure that the Control component does not leak any con dential information
about the operating plan that it receives from the Scheduler.
        </p>
        <p>
          In general, such information ow properties are veri able on the theoretical
basis of noninterference [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ], which states that a system is secure in terms of
information ow, if the con dential inputs to a system do not a ect any outputs
with a lower con dentiality. If a system ensures this policy, no illegitimate ow of
con dential information can ever occur. Whereas Barbuti and Tesei [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ] adjust the
theory of noninterference to real-time systems, their approach lacks a veri cation
technique that is amenable to industrial-scale design models, and is ready to use
by software engineers. This is in accordance with the observation by Mantel [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]
that \software engineering does not respect information ow security".
        </p>
        <p>
          A model-driven software design approach supporting the veri cation of
information ow properties is UMLsec [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ]. The authors use adversary models to
describe the capabilities of attackers, and verify that the design prevents
vulnerabilities that might be exploited by such attackers. Ochoa et al. [
          <xref ref-type="bibr" rid="ref15 ref16">16,15</xref>
          ]
discuss explicitly the noninterference veri cation of UMLsec statemachine models.
Houmb et al. [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ] provide guidelines to ensure traceability of security requirements
(challenge C3) through a systematic derivation of a UMLsec design. However,
due to the missing notion of real-time, the UMLsec approach is not capable of
verifying information ow properties of cyber-physical systems (challenge C2).
        </p>
        <p>In summary, none of the mentioned approaches supports the speci cation of
illegitimate information ow in model-based systems engineering, as demanded
by challenge C1. With respect to challenge C2, veri cation techniques either
disregard the real-time behavior, or lack an integration with model-driven software
design. Due to these shortcomings, none of the mentioned approaches is
currently suitable for tracing information ow requirements between speci cation
and software design (challenge C3).</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Proposed Solution</title>
      <p>We choose MUML as the underlying software design method due to its
existing support for real-time, and due to our preliminary work on the approach
(cf. Sect. 4). Furthermore, we rely on Consens for the speci cation of
requirements due to its tight integration with MUML. More precisely, we provide the
following partial solutions to the traceability problem for information ow
requirements:
Speci cation of illegitimate information ow at the level of Consens
system models. To address challenge C1, we provide systems engineers with a
speci cation technique for information ow requirements, allowing them to
mark a ow between elements of a system, or its environment, as
illegitimate. The speci cation needs to take place in a form that enables a later
comparison against the actual information ow detected through veri
cation. Thus, by distinguishing illegitimate from legitimate ow, it is possible
to judge whether the information ow requirements are violated.
Deriving veri able information ow properties from the speci ed
requirements in an automatic fashion. As a contribution to challenge C3, the derived
properties relate the initial requirements to a MUML software design model,
and allow to verify the model's compliance. In order to produce meaningful
veri cation results, the derived properties need to preserve the semantics of
the initial requirements and, therefore, the derivation needs to interrelate
the Consens system model and the MUML software design model. It is
bene cial to infer the relation between these models automatically from the
traces of a model transformation.</p>
      <p>
        Real-time veri cation of noninterference properties on the basis of
software design models. In order to overcome challenge C2, i.e., to decide whether
a given MUML model ful lls the derived information ow properties, a
rigorous veri cation of the noninterference needs to be carried out. To this end, we
enrich the theoretical basis of real-time noninterference [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] by a ready-to-use
veri cation technique. To cope with the in nite, real-valued statespace, we
explicitly consider the applicability of existing veri cation techniques from
the area of real-time model checking [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
      </p>
      <p>Reinterpretation of the veri cation results to trace them back to the
initial requirements as a further contribution to challenge C3. Depending on
the complexity of the interrelations between MUML software design and
Consens system model, the veri cation results obtained so far are of little
signi cance, as they do not allow the engineers to draw immediate
conclusions about the initial information ow requirements. Therefore, in order
to give signi cance to the results, we automatically relate them back to
the requirements speci ed initially. Every speci ed requirement needs to be
marked as met (if the non-occurrence of information ow has been proved by
the veri cation), or as a violation (if the veri cation detected the occurrence
of an illegitimate ow). Again, the trace of an earlier model transformation
from Consens to MUML might contribute to the reinterpretation of the
veri cation results by resolving the interrelations between both models.</p>
    </sec>
    <sec id="sec-4">
      <title>Preliminary Work</title>
      <p>
        Up to now, our research e orts focused mainly on the veri cation of MUML
software design models (challenge C2) in order to provide an underlying back-end
for the envisioned traceability solution. In [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ], we propose the application of
realtime model checking techniques in a cyber-physical system context. In order to
ensure domain-speci c applicability, we translate temporal logic properties from
MUML to the input language of a real-time model checker, and the veri cation
results back to MUML. However, by focusing on general temporal logic
properties, the veri cation of information ow properties is beyond the scope of the
approach up to now.
5
      </p>
    </sec>
    <sec id="sec-5">
      <title>Expected Contributions</title>
      <p>By overcoming challenge C2, we contribute to the veri cation of noninterference
properties in terms of a ready-to-use, yet theorized veri cation technique for
cyber-physical systems. In addition, we contribute a speci cation technique for
illegitimate information ow in the context of model-based systems engineering
by overcoming challenge C1. Finally, addressing challenge C3 will provide a
general insight into the traceability of requirements across the boundary between
discipline-spanning speci cation and discipline-speci c veri cation. On the
practical side, traceability enables systems engineers to receive feedback as to whether
the speci ed requirements are met or violated by the software design.
Furthermore, we also provide software engineers with a technique to verify information
ow properties of design models in a cyber-physical systems context.
6</p>
    </sec>
    <sec id="sec-6">
      <title>Plan for Evaluation and Validation</title>
      <p>To evaluate our contributions, we conduct case studies on the engineering of
highly interconnected cyber-physical systems with strict information ow
requirements, e.g., for protecting business secrets or personal data in the Industrial
Internet. Our case studies traverse the engineering process including model-based
systems engineering and software design. We will de ne measurable quality
characteristics to validate that our work meets the challenges described in Sect. 1:
C1: We contrast the requirements speci cation at the systems engineering level
with the speci cation of equivalent properties at the software design level,
to demonstrate the reduced e ort and expertise.</p>
      <p>C2: We compare the real-time veri cation against a technique without a concept
of time to demonstrate the improvements with respect to the number of
violations identi ed and false positives avoided.</p>
      <p>
        C3: We provide an integrated view of systems engineering and software design
to validate that our solution enables a precise distinction between violated
requirements on the one hand, and requirements that are met on the other
hand.
We currently work towards reducing the veri cation problem (challenge C2) to a
re nement check for real-time systems [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. After preparing the veri cation
backend, we plan to address the requirements speci cation at the level of model-based
systems engineering (challenge C1), and the derivation of veri able properties
within the next year. Finally, we intend to carry out the nal integration of
systems engineering and software design in order to establish the desired
traceability solution (challenge C3). Our evaluation strategy enables us to carry out
a stepwise, incremental validation of our contributions.
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Alur</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Courcoubetis</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dill</surname>
            ,
            <given-names>D.L.</given-names>
          </string-name>
          :
          <article-title>Model-checking in dense real-time</article-title>
          .
          <source>Information and Computation</source>
          <volume>104</volume>
          (
          <issue>1</issue>
          ),
          <volume>2</volume>
          {
          <fpage>34</fpage>
          (
          <year>1993</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Barbuti</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tesei</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>A decidable notion of timed non-interference</article-title>
          .
          <source>Fundamenta Informaticae</source>
          <volume>54</volume>
          (
          <issue>2-3</issue>
          ),
          <volume>137</volume>
          {
          <fpage>150</fpage>
          (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Dorociak</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dumitrescu</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gausemeier</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Iwanek</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Speci cation technique Consens for the description of self-optimizing systems</article-title>
          .
          <source>In: Design Methodology for Intelligent Technical Systems</source>
          . Springer (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Gausemeier</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          , Schafer, W.,
          <string-name>
            <surname>Greenyer</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kahl</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pook</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rieke</surname>
          </string-name>
          , J.:
          <article-title>Management of cross-domain model consistency during the development of advanced mechatronic systems</article-title>
          .
          <source>In: ICED 09</source>
          . pp.
          <volume>1</volume>
          {
          <issue>12</issue>
          (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Gerking</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          , Schafer, W.,
          <string-name>
            <surname>Dziwok</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heinzemann</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Domain-speci c model checking for cyber-physical systems</article-title>
          .
          <source>In: MoDeVVa 2015</source>
          . pp.
          <volume>18</volume>
          {
          <issue>27</issue>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Goguen</surname>
            ,
            <given-names>J.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Meseguer</surname>
          </string-name>
          , J.:
          <article-title>Security policies and security models</article-title>
          .
          <source>In: IEEE Symposium on Security and Privacy</source>
          <year>1982</year>
          . pp.
          <volume>11</volume>
          {
          <fpage>20</fpage>
          .
          <string-name>
            <surname>IEEE</surname>
          </string-name>
          (
          <year>1982</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Gotel</surname>
            ,
            <given-names>O.C.Z.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Finkelstein</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>An analysis of the requirements traceability problem</article-title>
          .
          <source>In: ICRE '94</source>
          . pp.
          <volume>94</volume>
          {
          <fpage>101</fpage>
          .
          <string-name>
            <surname>IEEE</surname>
          </string-name>
          (
          <year>1994</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Heinzemann</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brenner</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dziwok</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , Schafer, W.:
          <article-title>Automata-based re nement checking for real-time systems</article-title>
          .
          <source>Computer Science | Research and Development</source>
          <volume>30</volume>
          (
          <issue>3-4</issue>
          ),
          <volume>255</volume>
          {
          <fpage>283</fpage>
          (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Houmb</surname>
            ,
            <given-names>S.H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Islam</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Knauss</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          , Jurjens, J.,
          <string-name>
            <surname>Schneider</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>Eliciting security requirements and tracing them to design</article-title>
          .
          <source>Requirements Engineering</source>
          <volume>15</volume>
          (
          <issue>1</issue>
          ),
          <volume>63</volume>
          {
          <fpage>93</fpage>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10. Jurjens, J.:
          <source>Secure systems development with UML</source>
          . Springer (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Kokaly</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Salay</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sabetzadeh</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chechik</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Maibaum</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          :
          <article-title>Model management for regulatory compliance</article-title>
          .
          <source>In: MiSE@ICSE 2016</source>
          . pp.
          <volume>74</volume>
          {
          <fpage>80</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Mantel</surname>
          </string-name>
          , H.:
          <article-title>Information ow and noninterference</article-title>
          .
          <source>In: Encyclopedia of Cryptography and Security</source>
          , 2nd Ed., pp.
          <volume>605</volume>
          {
          <fpage>607</fpage>
          . Springer (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Nejati</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sabetzadeh</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Falessi</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Briand</surname>
            ,
            <given-names>L.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Coq</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          :
          <article-title>A SysML-based approach to traceability management and design slicing in support of safety certication</article-title>
          .
          <source>Information &amp; Software Technology</source>
          <volume>54</volume>
          (
          <issue>6</issue>
          ),
          <volume>569</volume>
          {
          <fpage>590</fpage>
          (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14. Object Management Group:
          <source>OMG Systems Modeling Language 1</source>
          .4 (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Ochoa</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cuellar</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pretschner</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hallgren</surname>
            ,
            <given-names>P.A.</given-names>
          </string-name>
          :
          <article-title>Idea: Unwinding based model-checking and testing for non-interference on EFSMs</article-title>
          .
          <source>In: ESSoS 2015</source>
          . pp.
          <volume>34</volume>
          {
          <fpage>42</fpage>
          . Springer (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Ochoa</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , Jurjens, J.,
          <string-name>
            <surname>Cuellar</surname>
          </string-name>
          , J.:
          <article-title>Non-interference on UML state-charts</article-title>
          .
          <source>In: TOOLS 2012</source>
          . pp.
          <volume>219</volume>
          {
          <fpage>235</fpage>
          . Springer (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>