<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Towards the Use of Graph Summaries for Privacy Enhancing Release and Querying of Linked Data</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Benjamin Heitmann</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Felix Hermsen</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Stefan Decker</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Informatik 5</institution>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Information Systems RWTH Aachen University</institution>
          ,
          <addr-line>Ahornstr. 55, 52056 Aachen</addr-line>
          ,
          <country country="DE">Germany</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Linked Data has become an important standard to describe meta-data about open government data. At the same time, most government data is not released as Linked Data. One reason for this could be the di culty of applying privacy enhancing technologies such as di erential privacy and private information retrieval to Linked Data. We introduce the idea of graph summaries to function as a schema for Linked Data which is schema-less. This in turn can provide a conceptual bridge for applying di erential privacy and private information retrieval to Linked Data.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>Linked Data using the DCAT vocabulary is emerging as the de-facto standard for
publishing meta-data about data sets released by governments, especially in the
EU. At the same time, most of the data sets described by the DCAT meta-data
are not being published using Linked Data.</p>
      <p>
        This might be due to the fact that there currently is a lack of best practices
for publishing sensitive data sets as Linked Data. For tabular data, established
methods in the area of di erential privacy [
        <xref ref-type="bibr" rid="ref1 ref2">1, 2</xref>
        ] exist, which preserves the utility
of the data set while limiting the recoverability of personally identi able
information and avoiding the issue of de-anonymising the data set.
      </p>
      <p>
        While Aron [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] makes a suggestion for an approach to apply di erential
privacy to Linked Data, one of the main short-comings is that the approach requires
an a priori list of classes or properties to protect.
      </p>
      <p>
        This is due to one of the main obstacles for applying the idea of di erential
privacy to Linked Data: Linked Data is schema-less, meaning that one data set
can contain properties and classes from many vocabularies and ontologies. In
addition, a priori knowledge about the structure of a linked data set is usually
not possible. Due to this exible and dynamic nature of LD, applying di erential
privacy like in [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] requires inspecting a data source and listing the sensitive classes
and properties.
      </p>
      <p>In this paper, we will introduce graph summaries as a central enabler for
privacy enhancing release and querying of Linked Data. We argue that future best
practices for identifying and masking the sensitive parts of an LD data set will
incorporate some form of graph summarisation. In addition, we will explain why
graph summaries could be very useful in enabling private information retrieval
by distributing a query which can be ful lled by one data source to multiple
data sources.</p>
      <p>
        The remainder of this paper is structured as follows: In section 2 we will
introduce the background in regards to di erential privacy and private information
retrieval. Then in section 3 we will describe an approach for graph summaries
which was introduced by Campinas et al. in [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. Then in section 4, we describe
how such graph summaries could be used to aid with the anonymised release of
Linked Data in accordance with Aron [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. In section 5 we describe how private
information retrieval from federated SPARQL endpoints is enabled by graph
summaries. We then conclude the paper and discuss future work in section 7.
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>Background</title>
      <p>The European Data Portal1 is publishing descriptions of data sets using the
DCAT vocabulary. In particular, the DCAT-AP pro le is used on the site. In
terms of size, currently descriptions of approx. 600,000 data sets from around 70
data portals of 34 countries are hosted by the European Data Portal. Yet, only
1500 of these data sets are declared as explicitly using a form of RDF.</p>
      <p>This suggests a mismatch between the requirements for releasing open
government data and the tools and best practices available for publishing Linked
Data. In particular, this could point to the di culty of anonymising Linked Data
sets before releasing them.</p>
      <p>The release of anonymised, statistical data sets in tabular form has been
solved to the most part using approaches which can be subsumed under the
heading of di erential privacy, which we describe below.</p>
      <p>This is followed by a description of private information retrieval (PIR). PIR
becomes relevant in relation to Linked Data, as SPARQL queries can reveal a
lot about the intentions and interests of the query issuing party.
2.1</p>
      <sec id="sec-2-1">
        <title>Di erential privacy</title>
        <p>The goal of releasing an anonymised statistical data, is to release data of a
representative sample of a population in order to enable analysis of the common
properties of that population, while keeping the properties of individual records
secret.</p>
        <p>
          The two most common elements of di erential privacy approaches are
generalisation and suppression in order to reduce the speci city of the quasi-identi ers [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ].
In addition, a traditional and simple approach has been the randomization
method [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ] which can be explained as the modi cation of data at collection
time by adding some noise to the records. If the noise is uniformly distributed,
1 http://www.europeandataportal.eu/
the randomization method bene ts from the fact that general data set properties
(such as mean) stay invariant, while the perturbation of the records makes an
identi cation hard.
        </p>
        <p>Nevertheless there is a trade o between the utility of the data and the level
of anonymization. On one hand, a great perturbation range gives a good
randomization but also alters the records at a great extend. Clearly, the data become
less expressive. On the other hand, a low bias retains the relevant information,
but outlying records mostly remain unvaried and an attacker can identify these
easily.</p>
        <p>
          Aron [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ] describes an approach for di erential privacy for RDF data, by
adding noise to Linked Data without changing the statistical properties of the
data. However, he points out that two important issues are remaining. These are
the issues of how to identify the representation of an individual record in the
data set, and the issue of identifying which properties are of a sensitive nature
and need to be protected.
        </p>
        <p>As we describe in Section 4, graph summaries can provide a source of a
priori knowledge about a linked data set, and have the potential to automate
the application of di erential privacy to Linked Data.
2.2</p>
      </sec>
      <sec id="sec-2-2">
        <title>Private information retrieval (PIR)</title>
        <p>
          The main idea of Private Information Retrieval (PIR) is that the user requests
a particular element of the database without the database owner knowing which
element the user was interested in [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ]. There is a distinction between
singleand multi-server PIR. In the single-server setup, the information is stored in
only one place whereas, in the multi-server setup, multiple databases contain
the same information thus allowing the user to load di erent parts of the data
from various servers.
        </p>
        <p>The main obstacle for traditional (non-SPARQL) multi-server PIR schemes,
is that all instances of the database would need to contain the same information.
This traditionally did imply the same organisation is in control of all the database
instances with its data.</p>
        <p>
          However, this scenario makes collusion of all database instances trivial.
Because of this, more focus was put on research of the single-server PIR scenario.
One of the results was an approach [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ] which is fast enough to stream binary
les and can saturate a 100Mbits/s line using a contemporary laptop.
        </p>
        <p>
          In contrast, for federated SPARQL queries the assumption is that each SPARQL
endpoint is controlled by a di erent organisation. This would make collusion
much more unlikely. However, on the other hand, the data which is stored in a
SPARQL endpoint is much more dynamic [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ].
        </p>
        <p>PIR becomes relevant in relation to Linked Data, as SPARQL queries can
reveal a lot about the intentions and interests of the query issuing party. If we
imagine a hypothetical use case in which open government data is hosted only
on the server of a government agency, and made only available via a SPARQL
endpoint, then that government agency can monitor all queries to the SPARQL
endpoint. If the data were available \in bulk", e.g. as an NTriples le, then all
the privacy concerns can be trivially addressed by just downloading the data le
and performing all queries locally. However, a malicious data provider will make
sure that the data is not available for download or missing important parts.</p>
        <p>In such a use case, the government could fully monitor all queries, e.g. made
by an NGO or a journalist, as only the SPARQL endpoint provided by the
government agency contains the data required to answer the query.</p>
        <p>As we discuss in Section 5, the availability of graph summaries for SPARQL
endpoints has the potential to enable splitting up a query which can be answered
by one SPARQL endpoint to multiple SPARQL endpoints. In our use case, this
would enable an NGO or a journalist to hide the details of their query and this
their intent from the organisations hosting the SPARQL endpoints.
3</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Graph summaries</title>
      <p>As explained in section 2, in order to apply privacy enhancing approaches to
Linked Data, such as di erential privacy or private information retrieval, the
contents of the Linked Data sets needs to be known in advance. This is due to
the fact that Linked Data is schema-less and fast changing.</p>
      <p>Graph summaries can provide a description of the graph, which contains the
structure, i.e. the types of links and the classes of entities. As such, it can take
the role of a schema as it is used for instance in relational databases. In addition,
a graph summary has the bene t of not containing any of the actual information
from the graph it summarises, if we assume that the class names and link names
contain no information.</p>
      <p>
        Campinas et al. [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] introduce a framework for graph summaries. The
framework provides di erent approaches to summarise a graph, which focus on either
types, properties or the structure. Figure 1 shows an example of a graph
summary focusing on the properties.
      </p>
      <p>
        Campinas et al. [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] present algorithms for both precise and approximate graph
summaries. Both types of summaries provide a graph which is homomorphic to
the original graph. However, while precise graph summaries contain every graph
which is also contained in the original graph, approximate graph summaries are
more robust in the face of graphs with errors and inconsistencies.
      </p>
      <p>In addition, for precise graph summaries the worst case for the size is to be
almost as big as the original graph. In contrast, approximate graph summaries
are much smaller.</p>
      <p>
        The steps of the algorithm are as follows, more details are in [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
Gathering of entity descriptions: The description of all entities is collected
with one complete pass over all edges of the graph. This provides the
necessary contextual information needed for the next step.
      </p>
      <p>Mapping of all nodes to their respective nodes in the summary graph
For every node, a corresponding summary node is either created or selected
from the list of already generated summary nodes.</p>
      <p>Materialisation of summary edges Another pass over all edges of the
original graph, in order to decide if that edge needs to be materialised in the
summary graph.</p>
      <p>
        After providing formal de nitions for both types of summaries, two
implementations for calculating graph summaries are given in [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. The rst version
uses SPARQL queries, while the second version can be run on shared-nothing
computation platforms such as Hadoop.
      </p>
      <p>
        Campinas provides the results of an evaluation on over 14 real-world datasets
of various size and complexity in Section 5.3 of [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. The results of the evaluation
show that Campinas algorithms make it possible to approximate the errorless
summary graph quite accurately but with a much lower space and time
complexity. In addition, the evaluation supports the claim that the algorithm can be
used on any kind of Linked Data.
      </p>
      <p>
        In comparison to other existing graph summarisation approaches, the
approach presented by Campinas in [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] and [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] is exible and expressive using
graph homomorphisms, whereas other approaches are designing for speci c
applications. No details of the graph need to be known beforehand, as the algorithm
is solely based on the features of the entities. The algorithm requires multiple
iterations, so the performance is dependant on the size of the graph.
      </p>
      <p>In summary, the approach for graph summaries proposed by Campinas et al.
can provide a schema for the schema-less Web of Data, while also having the
bene t of not revealing any other properties of a data set.</p>
    </sec>
    <sec id="sec-4">
      <title>Using graph summaries for the release of sensitive</title>
    </sec>
    <sec id="sec-5">
      <title>Linked Data</title>
      <p>Graph summaries can provide a source of a priori knowledge about a linked data
set, and have the potential to automate the application of di erential privacy to
Linked Data.</p>
      <p>
        As described by Aron [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], in order to apply di erential privacy to Linked Data,
it is necessary to know how to identify the enitites which need to be protected.
The availability of a graph summary allows heuristics to be now used to identify
such entities, based on existing de nitions of k-anonymity, l-diversity and
tcloseness [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], which we can paraphrase as follows for the graph summary:
k-anonymity A graph satis es k-anonymity, if for every instance of a class C,
there are k-1 other instances, so that their quasi-identi able attributes have
equal values with the other instances of class C.
l-diversity A graph satis es l-diversity, if for every class C, there are at least l
values for every property of every instance of class C.
t-closeness A graph satis es t-closeness if the distance between the distribution
of every property of every class C in the original graph and the anonymised
graph is not bigger than a threshold t.
      </p>
      <p>While the use of graph summaries has potential to allow the whole process of
anonymising and masking a linked data graph to be automated, there are many
reasons to still involve human inspection of the nal data set. In particular, to
prove that the responsibilities of an organisation have been ful lled.
5</p>
    </sec>
    <sec id="sec-6">
      <title>Using graph summaries for private information retrieval</title>
      <p>The availability of graph summaries for SPARQL endpoints has the potential to
enable splitting up of queries which can be answered by one SPARQL endpoint
to multiple SPARQL endpoints. This would enable an NGO or a journalist to
hide the details of their query and this their intent from the organisations hosting
the SPARQL endpoints.</p>
      <p>In particular, with the availability of graph summaries for all available SPARQL
endpoints, we can imagine the role of the query planner to be almost the
opposite of query planners which try to optimise the query response time. For private
information retrieval, the query planner will attempt to distribute a query across
as many SPARQL endpoints as possible. The intended goal then is to make it
impossible for any observer with only partial knowledge of the query to determine
the goal of the query. This mode of operation could also be called an obfuscating
query planner.</p>
    </sec>
    <sec id="sec-7">
      <title>Analysis of threat model</title>
      <p>
        Deng et al. [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] introduced a framework for analysing threat models. Using their
classi cation of threats, our suggestions for using graph summaries have to be
classi ed as \hard privacy" approaches, as they are limiting the release of data.
      </p>
      <p>
        First we list how the use of graph summaries for di erential privacy addresses
the hard privacy threats as de ned by Deng et al. in [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]:
Linkability: Using graph summaries allows removing of personally identi able
data, which mitigates the threat of linking an entity in the anonymised graph
with an entity in the original RDF graph.
      </p>
      <p>Identi ability: As entities in the original and anonymised graph are not
linkable, the identity of persons is also protected.</p>
      <p>Non-repudiation: In addition, as identities are protected, there should be no
proof of a persons information being part of the anonymised data set.
Detectability: In a similar way, participation of a person in the anonymised
data set should be undetectable.</p>
      <p>Disclosure of information: Finally, as this approach removes sensitive data
before it is released, disclosure of sensitive information is not possible.</p>
      <p>In summary, using graph summaries for di erential privacy maintains
unlinkability and anonymity of identities in the original data set, and it limits
unintended disclosure of information. However, it does not address non-repudiation
and detectability.</p>
      <p>Next we list how the use of graph summaries for private information retrieval
address hard privacy threats:
Linkability: Using an obfuscating query planner makes the di erent parts of
the query unlinkable to each other, provided there is no collusion between
the di erent SPARQL endpoints.</p>
      <p>Identi ability: Suitable credentials will be required in order to access all
required SPARQL endpoints, if no anonymous access without credentials is
available. Therefore the person or organisations initiating the queries is
always known.</p>
      <p>Non-repudiation: As standard SPARQL access is used, it is not possible to
deny having initiated the SPARQL queries.</p>
      <p>Detectability: In addition, as SPARQL queries are not sent over an encrypted
channel, any potential eavesdropper can listen to the SPARQL queries.
Disclosure of information: The information which is hidden from SPARQL
endpoints is the intent of the initiator and the original SPARQL query.
However eavesdroppers of the SPARQL queries could try to re-assemble the
original SPARQL query.</p>
      <p>In summary, using graph summaries for private information retrieval, hides
the intent behind the split SPARQL queries as the queries are unlinkable if
the SPARQL endpoint operators do not collude. In addition, the original query
and its intend are not disclosed. However, the privacy threats of identi ability,
non-repudiation and detectability are not addressed.</p>
    </sec>
    <sec id="sec-8">
      <title>Conclusion and future work</title>
      <p>We have introduced the idea of graph summaries as a privacy enhancing
technology to enable di erential privacy and private information retrieval.</p>
      <p>As graph summaries contain no information about a graph beyond the classes
and properties of the original data set, there is no leakage of personally
identi able information. Graph summaries can be used almost like the schema for
traditional, relational databases. As such they have the potential to provide a
conceptual bridge for applying many existing privacy enhancing technologies
from relational databases to Linked Data.</p>
      <p>In terms of future work, we are planning to implement the graph summary
approach as part of a framework for automatic anonymisation of Linked Data
sets. In particular, we will develop a testbed for evaluating di erent
parameterisations of anonymised release of Linked Data, against several state of the art
algorithms for deanonymising graph data as described in the overview of Al
Azizy et al. [12]. In addition, we are planning to implement a query processor for
obfuscated query planning to enable private information retrieval using graph
summaries.
12. Al-Azizy, D., Millard, D., Symeonidis, I., O'Hara, K., Shadbolt, N.: A literature
survey and classi cations on data deanonymisation. In: International Conference
on Risks and Security of Internet and Systems, Springer (2015) 36{51</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Venkatasubramanian</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Measures of anonymity</article-title>
          .
          <source>In: Privacy-Preserving Data Mining</source>
          . Springer (
          <year>2008</year>
          )
          <volume>81</volume>
          {
          <fpage>103</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Dwork</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Di erential privacy: A survey of results</article-title>
          .
          <source>In: International Conference on Theory and Applications of Models of Computation</source>
          , Springer (
          <year>2008</year>
          )
          <volume>1</volume>
          {
          <fpage>19</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Aron</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>Information privacy for linked data</article-title>
          .
          <source>Master's thesis</source>
          , MIT CSAIL (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Campinas</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Delbru</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tummarello</surname>
          </string-name>
          , G.:
          <article-title>E ciency and precision trade-o s in graph summary algorithms</article-title>
          .
          <source>In: Proceedings of the 17th International Database Engineering &amp; Applications Symposium</source>
          , ACM (
          <year>2013</year>
          )
          <volume>38</volume>
          {
          <fpage>47</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Yekhanin</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Private information retrieval</article-title>
          .
          <source>Communications of the ACM</source>
          <volume>53</volume>
          (
          <issue>4</issue>
          ) (
          <year>2010</year>
          )
          <volume>68</volume>
          {
          <fpage>73</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Aguilar-Melchor</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Barrier</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fousse</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Killijian</surname>
            ,
            <given-names>M.O.</given-names>
          </string-name>
          :
          <article-title>Xpir: Private information retrieval for everyone</article-title>
          .
          <source>Proceedings on Privacy Enhancing Technologies</source>
          <year>2016</year>
          (
          <article-title>2) (</article-title>
          <year>2015</year>
          )
          <volume>155</volume>
          {
          <fpage>174</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Buil-Aranda</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hogan</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Umbrich</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vandenbussche</surname>
          </string-name>
          , P.Y.:
          <article-title>Sparql webquerying infrastructure: Ready for action</article-title>
          ? In: International Semantic Web Conference, Springer (
          <year>2013</year>
          )
          <volume>277</volume>
          {
          <fpage>293</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Campinas</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Making Sense of Web Data</article-title>
          .
          <source>PhD thesis</source>
          , National University of Ireland, Galway (
          <year>2016</year>
          (to appear))
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Ciriani</surname>
          </string-name>
          , V.,
          <string-name>
            <surname>di Vimercati</surname>
            ,
            <given-names>S.D.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Foresti</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Samarati</surname>
          </string-name>
          , P.:
          <article-title>-anonymity</article-title>
          . In:
          <article-title>Secure data management in decentralized systems</article-title>
          . Springer (
          <year>2007</year>
          )
          <volume>323</volume>
          {
          <fpage>353</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Venkatasubramanian</surname>
          </string-name>
          , S.:
          <article-title>t-closeness: Privacy beyond k-anonymity and l-diversity</article-title>
          .
          <source>In: 2007 IEEE 23rd International Conference on Data Engineering</source>
          , IEEE (
          <year>2007</year>
          )
          <volume>106</volume>
          {
          <fpage>115</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Deng</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wuyts</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Scandariato</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Preneel</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Joosen</surname>
            ,
            <given-names>W.:</given-names>
          </string-name>
          <article-title>A privacy threat analysis framework: supporting the elicitation and ful llment of privacy requirements</article-title>
          .
          <source>Requirements Engineering</source>
          <volume>16</volume>
          (
          <issue>1</issue>
          ) (
          <year>2011</year>
          )
          <volume>3</volume>
          {
          <fpage>32</fpage>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>