<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Metamodel for Privacy Engineering Methods</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Yod-Samuel Martín</string-name>
          <email>samuelm@dit.upm.es</email>
          <email>samuelm@dit.upm.es ORCID: 0000-0002-0065-5117</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>José M. del Álamo</string-name>
          <email>jm.delalamo@upm.es</email>
          <email>jm.delalamo@upm.es ORCID: 0000-0002-6513-0303</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Departamento de Ingeniería de Sistemas Telemáticos, Universidad Politécnica de Madrid</institution>
          ,
          <addr-line>Madrid</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Departamento de Ingeniería de Sistemas Telemáticos, Universidad Politécnica de Madrid</institution>
          ,
          <addr-line>Madrid</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>- Engineering privacy in information systems requires systematic methods to capture and address privacy issues throughout the development process. However, the diversity of both privacy and engineering approaches, together with the specific context and scope of each project, have spawned a plethora of privacy engineering methods. Method engineering can help to cope with this landscape, as it allows describing existing methods in terms of a limited variety of method elements (and eventually enable their recombination into new, customized methods). This paper applies method engineering to introduce a privacy engineering metamodel, whose applicability is illustrated with a set of popular privacy engineering method elements, and a widely recognized privacy engineering method.</p>
      </abstract>
      <kwd-group>
        <kwd>Privacy engineering metamodel</kwd>
        <kwd>Method engineering</kwd>
        <kwd>Privacy engineering</kwd>
        <kwd>Privacy Methods</kwd>
        <kwd>Methodology</kwd>
        <kwd>Metamodel</kwd>
        <kwd>ISO/IEC 24744</kwd>
        <kwd>SEMDM</kwd>
        <kwd>Privacy by Design</kwd>
        <kwd>GDPR</kwd>
        <kwd>LINDDUN</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>I. INTRODUCTION</title>
      <p>
        Despite the increasing urgency in addressing privacy
concerns associated with information systems, and the
technical developments available, engineering privacy-friendly
systems remains a challenge for several reasons. First, privacy
is a multi-disciplinary, essentially contested concept [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], which
can thus be subject to multiple reference frameworks, be them
social, legal, or technical. Second, research efforts have
focused on tackling privacy issues by technical means, rather
than investing in generalizing and systematizing the application
of said technical solutions so that others can reuse and apply
them. Third, even when a given privacy framework is set, the
diversity of information systems (platforms, APIs, services,
infrastructures, enterprise systems…) and development process
models (agile, waterfall...) makes it difficult to elaborate a
onesize-fits-all privacy engineering method1.
      </p>
      <p>In this context, dozens of novel contributions in the field of
privacy engineering appear every year (of which the papers
presented at this workshop represent a sample), each of which
targets specific aspects and suits different situations. In order to
assess the benefit and adequacy of any such solution, it would
be desirable to have the relevant knowledge systematically
organized so as to ease the communication within the
community of practice and research of privacy engineering.</p>
    </sec>
    <sec id="sec-2">
      <title>1 For our purposes, we use both terms ‘method’ and ‘methodology’</title>
      <p>interchangeably.</p>
      <p>This work has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement No 731711.</p>
      <p>This paper describes our contribution to this effort, by
presenting a conceptual framework which allows arranging the
different concepts that usually underlie the various
contributions subsumed under the field of Privacy Engineering.
This framework has been realized as a metamodel which
extends SEMDM (the metamodel for software and systems
development methodologies described in ISO/IEC 24744
[ISO24744]), and provides a controlled vocabulary of privacy
engineering methodological elements and a normalized set of
connection points and relationships to organize those elements.
Thus, it enables the description of different elements of existent
privacy engineering methods in comparable terms, so that they
can be further catalogued and assessed. That metamodel can be
thought of as a labelled rack, to each of whose compartments
the contributions on privacy engineering can be anchored.
Moreover, by enriching the description of method elements
with well-defined connection hooks, their reuse and integration
is fostered.</p>
      <p>The remaining of the paper is structured as follows. Section
II provides some background on privacy and method
engineering. Then, section III describes our proposal for a
privacy engineering metamodel based on the extension of the
SEMDM metamodel, and section IV validates its applicability
by constructing a representation of LINDDUN, a well-known
privacy engineering method. Finally, section V discusses the
potential applicability of the metamodel we propose so as to
promote the reuse of privacy methodologies, and section VI
concludes by summarizing the significance of our solution and
pointing towards future work to overcome its limitations.</p>
    </sec>
    <sec id="sec-3">
      <title>II. BACKGROUND</title>
      <sec id="sec-3-1">
        <title>A. Privacy engineering</title>
        <p>
          Privacy engineering is a nascent field of research and
practice which pursues systematic approaches for the inception
and application of privacy-oriented solutions throughout
systems and software development processes. According to
one of the first definitions of privacy engineering [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ], the
keystones of the field are:
        </p>
        <p>Theories, which deal with privacy from different
approaches. For instance, for different authors, privacy
may be a matter of non-intrusion, seclusion, limitation,
control, boundary regulation, system architecture,
policy or interaction, just to mention a few. Following
that line, we consider that all privacy theories are born
valid to apply privacy engineering, but different
theories provide different conceptual frameworks to
•
•
•
which the elements of each privacy-engineering
method adhere.</p>
        <p>Methods, that is, processes for capturing and
addressing privacy concerns during any of the stages
of the lifecycle of information-based systems, which
include their conception, development, management,
and maintenance. Methods provide directions and rules
and help set privacy goals, structured in a systematic
way into tasks and stages with the aid of supporting
tools and techniques.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Techniques, which refer to procedures, possibly with a</title>
      <p>prescribed language or notation, to accomplish specific
privacy engineering tasks.2</p>
    </sec>
    <sec id="sec-5">
      <title>Tools, that is, means (automated or not) that support</title>
      <p>privacy engineers in carrying out their responsibilities
within a privacy engineering method.</p>
      <p>
        Efforts on privacy engineering usually stick to the “Privacy
by Design” (PbD) paradigm [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] which summons engineers and
other stakeholders to integrate privacy aspects into the different
activities they are involved, throughout the whole development
lifecycle of information-based systems, rather than introducing
them as an afterthought. Several PbD methods have been
developed which define engineering activities that introduce
privacy at different stages of the development lifecycle,
defining what are usually named whole-lifecycle privacy
methodologies.
      </p>
      <p>
        One such effort exemplifies all the concepts described
above: LINDDUN [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] is a privacy-engineering method focused
on the privacy assessment of information systems. It
conceptualizes privacy as seven distinct properties widely
recognized by the privacy research community, represented by
its corresponding threats (from whose initials LINDDUN takes
its name). This method describes a set of techniques to e.g.
identify privacy threats, and provides a tool called “threat
catalogue” that supports privacy engineers on this process (v.
section 4 below for a detailed description of LINDDUN in
terms of our privacy engineering metamodel).
      </p>
      <p>Although this conception of privacy engineering seems
clearly founded, there is no common standard framework
which privacy engineering developments may refer to. As a
matter of fact, efforts on standards for privacy have long been
undertaken by ISO/IEC JTC1/SC27/WG5 (Joint Technical
Committee 1 of the ISO and the IE, subcommittee 27 on IT
Security Techniques, working group 5 WG5 on identity
management and privacy technologies), which has delivered
general references that engineers and managers addressing
privacy must take into account with regards to terminology,
institutionalization of practice (i.e. ensuring that organizations
apply the same good practices), and support for evaluation (i.e.
approaches on how privacy is evaluated). However, those
efforts only provide partial views of privacy engineering, as
they deal with individual perspectives (e.g. privacy principles,
2 Note how these Techniques are methodological rather than technological,
and hence they are different from the “Privacy Enhancing Technologies”,
which belong to the realm of the technology applied at each endeavor.
best practices, organizational maturity), isolated processes (e.g.
impact assessment, requirements analysis), or specific domains
(Big Data, Internet of Things). Yet they lack a shared,
allencompassing conceptual framework, independent from
specific privacy engineering methodologies, development
practices and application domains.</p>
      <p>It is under these circumstances that we introduce our
approach, which proposes the definition of such a framework
where the different privacy engineering methodologies may be
pegged out, by formalizing the definition of a metamodel
which consists of the elements that usually appear in privacy
engineering methodologies and the relationships between one
another.</p>
      <sec id="sec-5-1">
        <title>B. Method engineering and SEMDM</title>
        <p>
          Our approach is grounded in the discipline of method
engineering, which is [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ] “the engineering discipline to design,
construct and adapt methods, techniques and tools for the
development of information systems”; and which focuses on
designing methods (or methodologies) for specific situations
(e.g. a specific organization, a specific project) rather than
resorting to rigid, existent methodologies.
        </p>
        <p>Any methodology that may be constructed responds to an
underpinning metamodel, i.e. an abstract model that describes
the concepts that may be present in the methodology (i.e. the
types of elements it is made of) and their potential relations
with one another. Many methodologies may exist that conform
to the same, shared metamodel (i.e. their descriptions can rely
on a common set of terms).</p>
        <p>
          One such metamodel is defined by the Software
Engineering Metamodel for Development Methodologies
(SEMDM)[
          <xref ref-type="bibr" rid="ref7">7</xref>
          ], standardized as ISO/IEC 24744 [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ], and “aimed
to the definition of methodologies in information-based
domains, i.e. areas characterized by their intensive reliance on
information management and processing, such as software,
business or systems engineering.”
        </p>
        <p>SEMDM proposes three layers of abstraction to define and
instantiate methodologies: metamodel, methodology and
endeavor (a.k.a. project). The metamodel defines the elements
that methods engineers employ to enact methodologies. In turn,
developers use the methodologies to construct products or
deliver services in the context of particular endeavors.</p>
        <p>The SEMDM metamodel describes a set of concepts that
can be part of any methodology, and which cover the three
dimensions of processes, producers (including people) and
products:
•
•</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>Work Units describe things to be executed, such as a</title>
      <p>Process (large-grained Work Unit that operates within
a given area of expertise), a Task (a small-grained
work unit that focuses on what must be done), or a
Technique (a small-grained work unit that focuses on
how it has to be done). A methodology may
recommend using specific techniques for a task, with
different degrees of Recommendation (e.g.
compulsory, optional, discouraged, etc.).</p>
      <p>Producers describe agents that have the responsibility
to carry out work units, and can be specialized into
•
•
•
•
•
•</p>
    </sec>
    <sec id="sec-7">
      <title>Roles (a collection of responsibilities that a producer can take), Tools (an instrument that helps another producer to execute its responsibilities in an automated way) and Teams (a set of producers).</title>
      <p>Work Products are artefacts of interest for the project
which can be used as inputs, intermediate results, or
outputs of a work unit; e.g. Document, Software Item
or Model. In SEMDM, a Model provides an abstract
representation of some modelling elements by
aggregating a set of Model Units, which may be related
with one another according to the grammar defined in
a Language. Work products may be acted upon by
work units through different Actions (read, create,
modify, or delete).</p>
    </sec>
    <sec id="sec-8">
      <title>Stages represent a managed time frame with a specific</title>
      <p>objective within a project, either instantaneous such as
a Milestone, or with duration such as a Phase (during
which the same cognitive framework prevails), a Build
(aimed delivering a version of a work product), or a
Time Cycle (aimed at delivering the final product or
service).</p>
      <p>
        A specific methodology (or a method fragment) will define
its own set of Work Units, Producers, Work Products, or
Stages (e.g. a method may define a type of Document called
Requirements Specification to be produced when the method is
enacted). Each of these elements defined in a methodology
holds a dual reality: on the one hand, they are instances of the
concepts defined in the metamodel (e.g. Requirements
Specification is a kind of Document); on the other one, they are
Templates that will be instantiated at each endeavor when the
methodology is enacted (i.e. each project will fulfill its own
instance of Requirements Specification, according to the said
Template). The relation between both perspectives is called a
‘powertype’ relationship. (The interested reader can get further
details on the use of the powertype pattern for methodology
metamodeling from [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].)
      </p>
      <p>SEMDM also defines Resources, that is, methodology
elements that are used ‘as is’ at the project level, without
requiring any instantiation, namely:</p>
      <sec id="sec-8-1">
        <title>Languages which define a set of Model Unit Kinds</title>
        <p>focused on one modelling perspective, and the
relations allowed among Model Units of those kinds.
A Language can be any complex, structured system of
related symbols able to convey meaning (which
encompasses the formal languages that underlie
programming languages, visual languages and natural
languages alike, but also conceptual languages as in
‘the language of art’).</p>
      </sec>
    </sec>
    <sec id="sec-9">
      <title>Notations which are associated to a Language, and provide a concrete syntax to represent Models conforming to that language.</title>
    </sec>
    <sec id="sec-10">
      <title>Guidelines which tell how to use some methodology elements.</title>
    </sec>
    <sec id="sec-11">
      <title>Constraints, that is, conditions that hold or must hold</title>
      <p>at certain point in time within the methodology.</p>
      <p>Thus, SEMDM provides a comprehensive and extendable
metamodel for system and software method engineering. We
have leveraged these features to develop a comprehensive
privacy engineering metamodel (detailed in the next section)
able to cope with the variety in privacy frameworks, business
domains, types of systems, and development processes.</p>
    </sec>
    <sec id="sec-12">
      <title>III. PRIVACY ENGINEERING METAMODEL</title>
      <p>If privacy engineering promoters want it to be actually
adopted, they cannot aim at proposing completely new
methodologies from scratch which are disconnected from
current practice. Rather, privacy engineering needs to be
aligned with more general efforts on software and systems
engineering, in order to ensure a smooth integration and ease
its application and utility.</p>
      <p>An important step in this direction would be the
formulation of a conceptual framework that focuses on
generalizing and systematizing privacy engineering
methodology elements, so that they can be compared, assessed,
and integrated.</p>
      <p>Method engineering (and SEMDM in particular) provides
an especially suitable foundation to model the field of privacy
engineering. Indeed, the aforementioned definition of privacy
engineering matches quite well the SEMDM metamodel, as
both consider the notions of theories, methods, techniques and
tools; to which SEMDM adds other concepts such as tasks,
stages, roles, teams and work units, which are also relevant to
privacy engineering methods, as we will show later. Besides,
SEMDM addresses both the process and the product
dimensions, both which are tackled by privacy engineering
methodologies.</p>
      <p>In consequence, we have built on and extended SEMDM to
propose a Privacy Engineering Metamodel, by identifying a set
of extensions to the standard SEMDM metamodel that support
the concepts specific to privacy engineering. The standard
SEMDM metamodel together with our extensions can be used
to describe any privacy-engineering methodology. Of course,
this can be further extended by specific privacy frameworks
which refine these concepts or define their own extensions.</p>
      <p>The most relevant extension to SEMDM that allows
dealing with privacy engineering aspects consists in the
definition of several types of Resources (in grey, Fig. 1)
present in many privacy engineering methods, and which
provide the foundations to deal with privacy engineering from
different perspectives, namely ontological (Privacy Conceptual</p>
      <sec id="sec-12-1">
        <title>Model), deontological (Privacy Normative Framework),</title>
        <p>situational (Privacy Engineering Code), and epistemological
(Privacy Knowledge Base). Besides, an abstract type of role
(Privacy Engineering Role) subsumes the common
responsibilities that may be expected from privacy engineers.</p>
      </sec>
      <sec id="sec-12-2">
        <title>A. Privacy Conceptual Model (PCM) and Units (PCUs)</title>
      </sec>
      <sec id="sec-12-3">
        <title>A Privacy Conceptual Model (PCM) provides a conceptual</title>
        <p>
          description of what ‘privacy’ is in the context of the privacy
theory where a specific privacy-engineering methodology is
grounded. Due to the plurality [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ], contextuality [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ], and
contestability [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ] of privacy as a social, political and legal
concept and its different translations to the technical domain,
we refrain from folding a specific conception of privacy into
the definition of our Privacy Engineering Metamodel. We do
however assume that any defendable privacy engineering
method will draw on some ontological definition of privacy,
which may in turn influence all the methodology elements.
        </p>
        <p>
          Besides providing a definition for privacy, the PCM may
answer other questions such as what its subject and object are
(cf. [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ]). For instance, ISO 29100 privacy framework [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]
defines what can be considered as personal information, which
actors can operate with it and in what interactions they can be
involved, etc.
        </p>
      </sec>
      <sec id="sec-12-4">
        <title>A PCM is often made of Privacy Conceptual Units (PCUs).</title>
        <p>
          For instance, some models [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ] specify privacy into a list of
privacy principles (fundamental, primary, or general guiding
rules for the implementation of privacy protections), others
[
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] as a set of privacy harms (problems that a data subject
may suffer as a consequence of an activity), yet some others
[
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] describe it as a set of technical goals (properties of the
system-to-be). Note how this partition of the concept of
privacy into conceptual units is not compulsory, e.g. some
theories [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ] conceive privacy without resorting to such
partition, —yet all respond to some given conceptual model.
        </p>
      </sec>
      <sec id="sec-12-5">
        <title>B. Privacy Normative Framework (PNF) and its components</title>
        <p>A Privacy Normative Framework (PNF) provides
normative requirements to be applied by all the methods
claiming to abide by it, and it may include binding regulations
as well as non-binding, recommended best practices. A PNF is
composed of three types of prescriptive elements:
1. Existential Constraints which require (or preclude) the
existence of specific elements in the methodology
(specific Tasks, Roles, Work Products, etc.).
2. Temporal Constraints 3 on the elements of a
methodology. They are expressed as entry or exit
conditions on Actions, which must hold at a certain
point in time (e.g. setting that an Action cannot be
executed unless the condition is met) —besides they
3 These are simply called Constraints by SEMDM, but we qualify them as
Temporal to distinguish them from Existential Constraints.
also affect indirectly the Work Products or Work Units
from the method.</p>
      </sec>
      <sec id="sec-12-6">
        <title>3. Privacy Endeavor Requirements (PER) are set on the</title>
        <p>system being developed in an endeavor. While both
Existential and Temporal Constraints apply to the
elements of the method itself, these Requirements
apply to the products created when the method is
enacted. Although a Requirement Set is typically
considered one of the Work Products produced during
an endeavor, in this case we are dealing with
highlevel requirements, which are provided by an external
Resource, to be obeyed ‘as is’ by the system.</p>
        <p>
          For example, the EU General Data Protection Regulation
(GDPR) [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ] provides a PNF that requires the existence
(Existential Constraint) of a Data Protection Officer (DPO)
Role with specifically allocated Tasks, prescribes that any
dataprocessing-related Task cannot be performed unless an impact
assessment Process has been carried out before (Temporal
Constraint), and mandates a set of Requirements to be met by
any system dealing with personal information. Note how the
specific PNF defined by GDPR commits as well to a given
PCM (viz. a set of principles relating to processing of personal
data and a set of rights of the data subject), but these are
different Resources even if referenced by the same source.
        </p>
      </sec>
      <sec id="sec-12-7">
        <title>C. Privacy Engineering Code (PEC)</title>
        <p>A Privacy Engineering Code (PEC) refines or clarifies the
application of the PNF under specific situations or contexts.
The PEC (sometimes known as ‘code of conduct’ or ‘code of
practice’) includes a set of Guidelines that document how a
Constraint or Requirement from the PNF can be applied
whenever a methodology is enacted on a specific context or
situation. The PEC may be typically subject to compliance or
audit checks. For example, Art. 40 of the EU GDPR
encourages that different institutions draw up codes of conduct
“…intended to contribute to the proper application of this</p>
      </sec>
      <sec id="sec-12-8">
        <title>Regulation, taking account of the specific features of the various processing sectors and the specific needs of micro, small and medium-sized enterprises.”</title>
      </sec>
      <sec id="sec-12-9">
        <title>D. Privacy Knowledge Base (PKB)</title>
        <p>A Privacy Knowledge Base (PKB) is a piece of generally
recognized knowledge that can be reused ‘as is’ in
privacyengineering endeavors, and whose value and usefulness are
collectively accepted. In our metamodel, a PKB is described as
a set of Model Units (instances of some kind of element
defined in a formal or conceptual Language), but which are
provided by a methodology as a Resource rather than created
by each endeavor. An analogy could be the set of standard
libraries provided by most programming languages alongside
the language specification itself, which define already
developed software components to be integrated with others
developed within an endeavor. A PKB can be used by
different types of Work Units defined in the methodology.</p>
        <p>
          Although privacy engineering is a nascent field, it has
already developed certain amount of generally recognized
knowledge, which is gathered in PKBs. For instance, privacy
patterns [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ] provide documented design solutions to common
privacy problems in particular contexts. Privacy patterns can
be described according to community-agreed templates and
pattern languages which define the relations among them, and
gathered together in privacy pattern repositories to be reused
by privacy engineers. Some other examples of currently
available PKBs are privacy design strategies [
          <xref ref-type="bibr" rid="ref18">18</xref>
          ] and privacy
threats [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ].
        </p>
      </sec>
      <sec id="sec-12-10">
        <title>E. Privacy Engineering Roles (PER)</title>
        <p>
          Some Privacy Engineering Role (PER) participates in one
way or another in most privacy engineering methods. It
represents someone who understands the privacy framework, is
aware of the privacy engineering methodology elements that
lead to the development of privacy-enhanced systems, and is
able to apply them within the endeavor at hand. As such,
Privacy Engineering Roles are characterized by their
multidisciplinarity, being savvy in the three of privacy,
engineering and the domain of the specific endeavor. As stated
by Cranor [
          <xref ref-type="bibr" rid="ref20">20</xref>
          ], “[a] privacy engineer is someone who
understands the engineering and the privacy sides and works
out strategies that allow people to protect privacy without
getting in the way of building cool things.”
        </p>
        <p>
          Note that the PER represents an abstract role which shall be
instantiated by more specific Roles defined by particular
privacy engineering methods (e.g. Privacy Requirements
Engineer, etc.), with the specific responsibilities set by the
methodology. For instance, the Carnegie Mellon University
M.Sc. In IT - Privacy Engineering enumerates a wide range of
responsibilities that may be assumed by privacy engineers [
          <xref ref-type="bibr" rid="ref21">21</xref>
          ]:
“[…]develop technical solutions to help mitigate privacy
vulnerabilities; analyze software designs and implementations
from a privacy and UX perspective; research, document, and
help remediate design decisions, operating procedures, or
processes that may directly or indirectly contribute to future
privacy risks; create cutting-edge privacy feature prototypes;
help to lead better on privacy by example; and partner with key
business, technical and legal stakeholders across various
business groups to implement Privacy by Design.”
        </p>
        <p>Besides these Privacy Engineering Roles, any privacy
engineering method may define additional, concrete roles (and
their associated responsibilities) that must be considered at the
methodology level. For instance, the EU GDPR identifies the
roles of: data protection officer (DPO) with responsibilities in
the privacy impact assessment, certification body with
responsibilities in the audit and/or certification process, or
independent supervisory authorities with responsibilities e.g. in
the consultation prior to the processing. These roles need not
represent privacy engineers (as they do not meet the aforesaid
threefold savvy), yet they are part of the privacy engineering
method (as they are involved in some of the tasks there
defined).</p>
        <p>IV. DESCRIPTION OF LINDDUN IN TERMS OF THE PRIVACY</p>
        <p>ENGINEERING METAMODEL.</p>
        <p>For the proposed Privacy Engineering Metamodel to be
useful, the concepts involved in privacy engineering
methodologies should be mappable to either SEMDM
methodology elements or to the extensions that we have
introduced above. In particular, as a validation of the
applicability of our metamodel, we have applied it to describe
the LINDDUN methodology (a well-known
privacyengineering method) and its main elements, as shown next.</p>
        <p>
          LINDDUN [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ] is a model- and knowledge-based privacy
engineering methodology aimed at systematically identifying
the privacy threats in a system and the solutions that mitigate
them, by following six linear steps, namely:
1.
2.
        </p>
      </sec>
    </sec>
    <sec id="sec-13">
      <title>Define a data flow diagram (DFD), departing from</title>
      <p>either the requirements specification or the system
architecture, while focusing on the internal data stores
and the data flows that cross the organization
boundaries, rather than on the internal processes.</p>
      <p>Map privacy threat categories to DFD elements (just
defined in the step above), according to a predefined
table that details potential threat categories for each
type of DFD element; while optionally discarding less
likely threats, and reducing threats with common
elements to a single one.
3. Identify threat scenarios, according to the guidance
provided by a set of privacy-threat-tree patterns,
describing threats in terms of misuse cases, and
documenting any assumptions made.
4. Prioritize threats, depending on the risk associated to
each one, according to the results of a risk assessment
external to this methodology.
5. Elicit mitigation strategies, according to a taxonomy of
strategies and a table that maps threat types to
strategies.
6. Select Privacy-Enhancing Technologies (PETs),
constrained by the mitigation strategies just elicited.</p>
      <p>It is not difficult to realize how LINDDUN methodology
can be modelled in terms of the elements of our Privacy
Engineering Metamodel (including both native SEMDM
elements and the extensions we have defined).</p>
      <p>Each of the LINDDUN steps specifies what must be done
in order to follow the methodology, that is, they define
different types of Tasks. Besides, most of these steps also detail
specific procedures to be followed in order to complete the
respective task, that is, they also define some associated
Techniques. These techniques are sometimes mandatory (e.g.
when threats are elicited, they must be refined using threat tree
patterns, documented according to a threat description template
together with any assumptions made), other times they are
merely recommended (e.g. strategies and solutions should be
respectively elicited using LINDDUN-provided mappings, but
these are a mere convenience), and others are optional (e.g.
DFDs can be created following specific techniques departing
from specifications or architecture, but other techniques can be
followed as long as the resulting DFD accurately models the
data flows in the system). In some cases, LINDDUN does not
even provide any technique for the respective task, but refers
the reader to external sources (e.g. threat prioritization depends
on applying techniques specified elsewhere, in order to
compute the likelihood and impact of privacy threats). The said
Tasks are also grouped into two Processes, namely the
elicitation of privacy threats (which covers the first three steps)
and the selection of mitigating solutions (the three last). These
same Processes shall be iterated throughout the development
cycle. And from the temporal perspective, these LINDDUN
Processes can be performed during different Phases of a
software and systems development methodology. Although the
specific phases shall depend on and align with the development
methodology employed, LINDDUN authors themselves
suggest that these Processes can be applied several times
during the “requirements” (i.e. inception) phase, during the
“architecture” (i.e. elaboration) phase, or during the
maintenance phase (on existing systems).</p>
      <sec id="sec-13-1">
        <title>These Work Units (Tasks, Techniques and Processes)</title>
        <p>
          produce tangible results, i.e. different types of Work Products.
More specifically, the DFD is a type of Model (hence the
description of LINDDUN as “model-based”), whose Model
Units (viz. external entities, data stores, data flows, and
processes) respond to a Language and are represented
(depicted) using a graphical Notation. Likewise, Misuse Cases
[
          <xref ref-type="bibr" rid="ref22">22</xref>
          ] are types of Models employed to describe threats. And
different types of Documents (the threat mapping table; and the
lists of threats, assumptions, mitigation strategies and PETs)
are created by instantiating the respective Templates. Besides,
Work Products can be not only created, but also modified or
merely read, e.g. when the outputs of a Work Unit are then
used as inputs by another one. LINDDUN even allows for
using Work Products that have been created elsewhere (e.g. the
requirements specification or the architecture from which the
DFD can be derived).
        </p>
        <p>Some of the methodology steps are further supported by
predefined catalogues of privacy threat trees, mitigation
strategies and privacy-enhancing solutions. That is, the
methodology provides three Privacy Knowledge Bases (once
again, hence the “knowledge-based” feature of the
methodology). These knowledge bases include each a list of
atomic components or Model Units (threats, strategies,
solutions), besides defining the relationships among them.
These PKBs can be used by any Producer that applies the
method, in order to simplify the elicitation processes by
directly including these Model Units as needed, rather than
coming up ex novo with other threats, strategies and solutions.</p>
        <p>Table 1 (below) models the concepts defined by
LINDDUN in terms of these method elements (Processes,
Privacy
Threat Tree
Catalogue
sPrioritize – –
inthreats
o
t
lsuo M Threat List
tingElicit R Map threats R Threat List Mitigation
itifaogmsmtriatitgeagtiieosn to strategies C LMiisttigation Strategies (S&amp;TtraMaxteaogpnipoeimsngy)
itcnoSPreilveaccty- R sMtraaptegies to R LMiisttigation Strategies PErnihvaancyci-ng
leeEnhancing solutions C PETs List Solutions
STechniques Catalogue
a. Recommended Usage: M = Mandatory, R = Recommended , O = Optional, D =
bc.. DAWicostcirokonuPrTraoygdpeudec,:tFCs=i=nCFitoraerlabictiesd,daMeren=noMt odedfiifnye,dDb=yDLIeNleDteD,RUN= Ritseealdf-bountlyelsewhere (nonetheless,
they are used by LINDDUN).</p>
      </sec>
      <sec id="sec-13-2">
        <title>Tasks, Techniques, Work Products and Resources) and</title>
        <p>provides the relations between them.</p>
        <p>All LINDDUN’s methodology elements are influenced by
its underlying Privacy Conceptual Model, which consists of
nine privacy properties (that is, Privacy Conceptual Units), viz.
unlinkability, anonymity, pseudonimity, plausible deniability,
undetectability, unobservability, confidentiality, awareness,
and compliance. LINDDUN Privacy Threats are accordingly
classified into seven categories (after whose initials LINDDUN
is named), depending on the property respectively
compromised: Linkability, Identifiability, Non-repudiation,
Detectability, Disclosure of information, user Unawareness,
and Non-compliance. It is through these categories that the
influence of the PCM pervades the LINDDUN methodology.
Thus, the structure of several Templates of Documents and</p>
      </sec>
      <sec id="sec-13-3">
        <title>Privacy Knowledge Bases in LINDDUN matches these</title>
        <p>categories, which guide as well the threat elicitation Process
that yields the Threat List. The latter is employed to elicit the
mitigation strategies, which in turn guide the selection of PETs,
hence both are also indirectly affected by the threat categories.
V. METHOD REUSABILITY AND INTEGRATION THROUGH THE</p>
        <p>PRIVACY ENGINEERING METAMODEL</p>
      </sec>
      <sec id="sec-13-4">
        <title>It shall be noted that our Privacy Engineering Metamodel</title>
        <p>does not prescribe that all privacy methodologies incorporate
all the types of Resources and other elements herein presented.
Rather, we merely describe those elements which appear
frequently in privacy methods, so as to offer a common
theoretical model. What is more, as we discuss in this section,
and siding with the objectives pledged by method engineering,
the definition of different methodologies in compatible terms
might ease the integration of elements or fragments from
different privacy methods (whether in whole or in part) with
one another and within generic (i.e. non-privacy-specific)
software and systems engineering methodologies. Hence a
specific privacy method may lack some of the metamodel
elements; however, this should not be considered a drawback,
but rather a feature, which matches the paradigm of method
fragment reusability.</p>
        <p>We anticipate that different privacy engineering methods
will coexist, which respectively suit better the needs of specific
fields, organizations, endeavors, legislations or technologies
(i.e. there will not be any overarching method that covers the
whole of privacy engineering). We assume that engineers may
be willing to leverage these methods, so as to benefit from their
usage beyond their initially planned scope, and integrating
elements picked from different methods. And we posit that the
definition of privacy methods in terms of the metamodel herein
presented may ease their reuse and integration.</p>
        <p>
          Indeed, the declared purpose of method engineering [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]
consists in enabling the assembly of methodologies from
method fragments coming from different sources, so as to
develop new methods that fit better the endeavor at hand.
SEMDM facilitates this methodological flexibility regarding
each specific situation, by introducing three distinct layers for
the metamodel, the methodology and the endeavor. Thus, two
methods might be more easily integrated as long as they are
described in compatible terms i.e. drawn from the same
metamodel (to avoid an apples-and-oranges situation). Yet the
metamodel not only provides a shared terminology, but also
defines a series of hooks or extension points where elements
from both methods can interface with each other. Then,
departing from existent method parts or elements (e.g. various
definitions of tasks, techniques, products), a methodologist can
design new methods through different strategies (e.g. by
assembling different fragments already available), always
taking into account the goals that the method under
construction is expected to fulfill. Therefore, method
engineering responds to the question of how a method is
developed in a context where relevance must be given to
specific domain constraints and goals that the method under
construction must fulfill (in our case, privacy constraints and
goals). The answer comes by tailoring the method (rather than
sticking to a predefined methodology set in stone) and
selecting appropriate method elements to the specific context
of each endeavor.
        </p>
        <p>For instance, and keeping at the LINDDUN example, it
does not prescribe any Privacy Normative Framework, nor
does it feature any Privacy Engineering Code. A methodologist
might anyway integrate LINDDUN with a specific PNF or</p>
      </sec>
      <sec id="sec-13-5">
        <title>PEC, by introducing the respective Constraints, Requirements</title>
        <p>and Guidelines as new method elements, and evaluating their
impact in LINDDUN-defined elements. Likewise, it happens
that LINDDUN is only focused on Work Units and Work
Products, but it is agnostic regarding the Producers that
perform and act upon them. Nonetheless, it should not be
difficult to map elements from other methodologies (e.g.
analysts, architects, etc.) to Producer ‘placeholders’.</p>
        <p>Furthermore, the Tasks defined by LINDDUN may depend
on the integration with external methods. For instance,
LINDDUN prescribes a Task to prioritize threats, which
depends on the result of a risk assessment Process. However,
neither does it tell what Technique to employ for that risk
assessment, nor does it define the Document Template for the
risk assessment document, leaving both to the implementer’s
decision. Other sources are available that provide techniques to
compute a privacy threat likelihood and its impact which can
be easily introduced, as long as they produce a list of risk
indices for each threat which can be used for prioritization.</p>
        <p>Moreover, LINDDUN does not even encompass all the
possible privacy-related Tasks either. It only deals with some
of them, while leaving out of its scope the creation of privacy
policies; privacy testing, assessment, and auditing, etc., which
could be specified by other methodologies. This can be eased
by mapping LINDDUN Tasks onto different Processes from
generic development methodologies (e.g. system analysis, risk
assessment, architecture engineering), and completing the
absent Processes with Tasks from other privacy-engineering
methods.</p>
        <p>All in all, LINDDUN elements would need to be integrated
with elements from other methodologies in order to cover the
whole development lifecycle, and they should be embedded
within a mainstream development methodology (e.g. Agile,
Unified Process, etc.) where privacy aspects would only play a
limited part. All the intervening methods should be first
modelled in terms of SEMDM so as to allow connecting their
elements with one another. And only then, the resulting,
composed methodology might be applied to new projects.</p>
      </sec>
    </sec>
    <sec id="sec-14">
      <title>VI. CONCLUSION</title>
      <p>It will be difficult that privacy engineering succeeds unless
there is a common, shared understanding of its underlying
concepts and the relationships between one another. We have
defined such a common conceptual framework: a Privacy
Engineering Metamodel that extends the SEMDM metamodel,
and which paves the way to reuse and assemble methodologies.
We have demonstrated the application of our metamodel by
decomposing LINDDUN into its constituent elements, defined
in terms of the SEMDM metamodel plus our extensions, and
suggesting how these elements might be reused and integrated
with other methodological approaches.</p>
      <p>Most relevant concepts for privacy engineering can be
mapped onto the elements provided by SEMDM,
supplemented with our extensions. It should be noted that we
do not claim either the novelty of these additional concepts
(some of which have long been established in the general
discipline of Engineering), or its exclusivity to the field of
privacy engineering (as most can be applied to other categories
of non-functional requirements as well); but just that we have
identified them as appropriate to model privacy engineering,
though they were not available straightaway from SEMDM. In
fact, these extensions might well be considered as relevant
contributions in the context of SEMDM itself.</p>
      <p>Despite the proven need, a potential problem of our
metamodel is the lack of a guarantee of adoption. It may
happen that some proponents of privacy engineering
methodologies refrain from recognizing other alternatives
deemed to be as valid as theirs. We understand that, even if
they cannot be reconciled, at least they may still agree on a
common conceptual metamodel. We also plan to refine our
metamodel by providing more examples of specific Work</p>
      <sec id="sec-14-1">
        <title>Units, Producers and Work Products, aligned to the</title>
        <p>taxonomies usually employed in the field of method
engineering (e.g. OPFRO), which demonstrate its applicability
to further privacy engineering methodologies.</p>
        <p>It may also be the case that our proposed metamodel is only
adopted within reduced academic circles, and contributes to
further fragmentation rather than preventing it. That is why we
aim to submit it to active standardization efforts. It is the case
that the ISO and the IEC have taken the same challenge as
well, by recently approving the work item (i.e. launching the
development of a standard) ISO/IEC AWI 27550 on Privacy
Engineering. This standard will aim to provide guidelines on
how to engineer privacy in information systems considering
different domains and under different development processes.
We aim at contributing our metamodel to this standard, whose
creation demonstrates the need to tackle the gap we are dealing
with. Making it a de jure standard will foster its de facto
adoption.</p>
        <p>The metamodel we have presented is the result from
extracting common features from several methodologies we are
acquainted with. Here we have presented a specific validation
case that exemplifies it (LINDDUN), but it may be well
applied to other methodologies. Not only the existence of a
shared conceptual model is itself key to foster the advance of
the discipline; but also its modularity allows adapting it to
specific endeavors or constraints. Rather than aspiring to come
up with a single catch-all privacy engineering methodology;
the modularity allows method engineers to create their own
methodologies. We conjecture that ultimately, if the approach
we propose succeeds, it will enable the integration of privacy
engineering methodological elements with one another and
within mainstream software and system development
methodologies, and ultimately, will improve the privacy of the
products or projects developed according to methodologies
based on our metamodel —which is something yet to be
proved in practice, so as to demonstrate the utility and
effectiveness of our approach. In any case, we expect other
members from the community to discuss the metamodel and
enrich it with their own contributions.</p>
        <p>It is the case, however, that the rigorous application of
method engineering principles has been in general limited (e.g.
to critical systems), and it remains a tool to organize the
knowledge rather than a practical way to integrate different
methodologies. Anyway, achieving the same result in the field
of privacy engineering would still be a success nonetheless.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>D. K.</given-names>
            <surname>Mulligan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Koopman</surname>
          </string-name>
          , and
          <string-name>
            <given-names>N.</given-names>
            <surname>Doty</surname>
          </string-name>
          , “
          <article-title>Privacy is an essentially contested concept: a multi-dimensional analytic for mapping privacy: Table 1</article-title>
          .,” Philos.
          <source>Trans. R. Soc. A Math. Eng. Sci.</source>
          , vol.
          <volume>374</volume>
          , no.
          <year>2083</year>
          , p.
          <fpage>20160118</fpage>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>S.</given-names>
            <surname>Gurses and J. M. del Alamo</surname>
          </string-name>
          , “Privacy Engineering:
          <article-title>Shaping an Emerging Field of Research</article-title>
          and Practice,” IEEE Secur. Priv., vol.
          <volume>14</volume>
          , no.
          <issue>2</issue>
          , pp.
          <fpage>40</fpage>
          -
          <lpage>46</lpage>
          , Mar.
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>A.</given-names>
            <surname>Cavoukian</surname>
          </string-name>
          , “
          <article-title>Privacy by Design The 7 Foundational Principles</article-title>
          ,” Toronto, Ontario (Canada),
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>K.</given-names>
            <surname>Wuyts</surname>
          </string-name>
          and
          <string-name>
            <given-names>W.</given-names>
            <surname>Joosen</surname>
          </string-name>
          , “
          <article-title>LINDDUN privacy threat modeling: a tutorial</article-title>
          .” Department of Computer Science, KU Leuven,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>S.</given-names>
            <surname>Brinkkemper</surname>
          </string-name>
          , “Method Engineering: Engineering of Information Systems Development Methods and Tools,” Inf. Softw. Technol., vol.
          <volume>38</volume>
          , no.
          <issue>4</issue>
          , pp.
          <fpage>275</fpage>
          -
          <lpage>280</lpage>
          ,
          <year>1996</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>B.</given-names>
            <surname>Henderson-Sellers</surname>
          </string-name>
          and
          <string-name>
            <given-names>J.</given-names>
            <surname>Ralyté</surname>
          </string-name>
          , “Situational Method Engineering:
          <article-title>State-of-the-Art Review</article-title>
          .,
          <source>” J. Univers. Comput. Sci.</source>
          , vol.
          <volume>16</volume>
          , no.
          <issue>3</issue>
          , pp.
          <fpage>424</fpage>
          -
          <lpage>478</lpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>C.</given-names>
            <surname>Gonzalez-Perez</surname>
          </string-name>
          and
          <string-name>
            <given-names>B.</given-names>
            <surname>Henderson-Sellers</surname>
          </string-name>
          ,
          <article-title>“A powertype-based metamodelling framework</article-title>
          ,” Softw. Syst. Model., vol.
          <volume>5</volume>
          , no.
          <issue>1</issue>
          , pp.
          <fpage>72</fpage>
          -
          <lpage>90</lpage>
          ,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <article-title>[8] ISO/IEC JTC 1</article-title>
          /SC 27,
          <string-name>
            <surname>“</surname>
            <given-names>ISO</given-names>
          </string-name>
          /IEC 24744:
          <year>2014</year>
          .
          <article-title>Software engineering -- Metamodel for development methodologies</article-title>
          ,”
          <source>Geneva (CH)</source>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>B.</given-names>
            <surname>Henderson-Sellers</surname>
          </string-name>
          and
          <string-name>
            <given-names>C.</given-names>
            <surname>Gonzalez-Perez</surname>
          </string-name>
          , “
          <article-title>On the ease of extending a powertype-based methodology metamodel,” in Meta modelling</article-title>
          and ontologies :
          <source>proceedings of the 2nd International Workshop on MetaModelling, WoMM 2006, October 12-13</source>
          , 2006 in Karlsruhe, Germany,
          <year>2006</year>
          , pp.
          <fpage>11</fpage>
          -
          <lpage>25</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>D. J.</given-names>
            <surname>Solove</surname>
          </string-name>
          , “Conceptualizing privacy,” California Law Review, vol.
          <volume>90</volume>
          , no.
          <issue>4</issue>
          . pp.
          <fpage>1087</fpage>
          -
          <lpage>1155</lpage>
          ,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>H.</given-names>
            <surname>Nissenbaum</surname>
          </string-name>
          , “
          <article-title>Privacy as contextual integrity,”</article-title>
          <string-name>
            <given-names>Wash. L.</given-names>
            <surname>Rev</surname>
          </string-name>
          ., pp.
          <fpage>101</fpage>
          -
          <lpage>139</lpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12] ISO/IEC JTC 1/SC 27, “Information technology --
          <string-name>
            <surname>Security</surname>
          </string-name>
          techniques - -
          <source>Privacy framework ISO/IEC</source>
          <volume>29100</volume>
          :
          <year>2011</year>
          ,” Geneva (CH),
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>D. J.</given-names>
            <surname>Solove</surname>
          </string-name>
          , “A Taxonomy of Privacy,” Univ. PA.
          <source>Law Rev.</source>
          , vol.
          <volume>154</volume>
          , no.
          <issue>3</issue>
          , pp.
          <fpage>477</fpage>
          -
          <lpage>560</lpage>
          ,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>A.</given-names>
            <surname>Pfitzmann</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Hansen</surname>
          </string-name>
          , “
          <article-title>A terminology for talking about privacy by data minimization: Anonymity, Unlinkability</article-title>
          , Undetectability, Unobservability, Pseudonymity, and Identity Management,”
          <source>Tech. Univ. Dresden</source>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>98</lpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>S.</given-names>
            <surname>Petronio</surname>
          </string-name>
          , Boundaries of Privacy: Dialectics of Disclosure. New York, New York, USA: State University of New York Press,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Regulation</surname>
          </string-name>
          (EU)
          <year>2016</year>
          /
          <article-title>679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data</article-title>
          ,
          <source>and repealing Directive</source>
          <volume>95</volume>
          /46/EC (General Da.
          <source>European Union: Official Journal of the European Union, L 119</source>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>88</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <article-title>“privacypatterns.eu - collecting patterns for better privacy</article-title>
          .” [Online]. Available: https://privacypatterns.eu/. [Accessed:
          <fpage>19</fpage>
          -Feb-2017].
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <surname>J.-H. Hoepman</surname>
          </string-name>
          , “Privacy Design Strategies,”
          <source>in ICT Systems Security and Privacy Protection SE - 38</source>
          , vol.
          <volume>428</volume>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Cuppens-Boulahia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Cuppens</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Jajodia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Abou El Kalam</surname>
          </string-name>
          , and T. Sans, Eds. Springer Berlin Heidelberg,
          <year>2014</year>
          , pp.
          <fpage>446</fpage>
          -
          <lpage>459</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>K.</given-names>
            <surname>Wuyts</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Scandariato</surname>
          </string-name>
          , and W. Joosen, “
          <string-name>
            <surname>LIND</surname>
          </string-name>
          (
          <article-title>D)UN privacy threat tree catalog</article-title>
          .” Department of Computer Science, KU Leuven,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>E.</given-names>
            <surname>Heil</surname>
          </string-name>
          , “
          <string-name>
            <surname>Q</surname>
          </string-name>
          &amp;
          <article-title>A: Privacy engineers could hold the key,” TribLIVE</article-title>
          .com,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <surname>“Privacy Engineering Careers-MSIT-Privacy</surname>
            <given-names>Engineering</given-names>
          </string-name>
          - Carnegie Mellon University.” [Online]. Available: http://privacy.cs.cmu.edu/careers/. [Accessed:
          <fpage>19</fpage>
          -Feb-2017].
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>G.</given-names>
            <surname>Sindre</surname>
          </string-name>
          and
          <string-name>
            <given-names>A. L.</given-names>
            <surname>Opdahl</surname>
          </string-name>
          , “
          <source>Templates for Misuse Case Description,” 7th Int. Work. Requir. Eng. Found. Softw. Qual. REFSQ</source>
          <year>2001</year>
          , vol.
          <volume>6</volume>
          , pp.
          <fpage>125</fpage>
          -
          <lpage>136</lpage>
          ,
          <year>2001</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>