In this paper, we propose generating artificial data that retain statistical properties of real data as the means of providing privacy for the original dataset. We use generative adversarial networks to draw privacy-preserving artificial data samples and derive an empirical method to assess the risk of information disclosure in a differential-privacy-like way. Our experiments show that we are able to generate labelled data of high quality and use it to successfully train and validate supervised models. Finally, we demonstrate that our approach significantly reduces vulnerability of such models to model inversion attacks.
Following recent advancements in deep learning, more and
more people and companies get interested in putting their
data in use and employ machine learning (ML) to generate
a wide range of benefits that span financial, social,
medical, security, and other aspects. At the same time, however,
such models are able to capture a fine level of detail in
training data, potentially compromising privacy of
individuals whose features sharply differ from others. Recent
research
The latter result is especially disturbing knowing that deep learning models are becoming an integral part of our lives, making its way to phones, smart watches, cars, and appliances. And since these models are often trained on customers’ data, such training set recovery techniques endanger privacy even without access to the manufacturer’s servers where these models are being trained.
One direction to tackle this problem is enforcing privacy
during training
Labels Noise
Critic Generator
Real
Fake Artificial Data
ML each of them. Moreover, most of these methods assume (implicitly or explicitly) access to public data of similar nature, which may not be possible in areas like medicine.
In contrast, we study the task of privacy-preserving data release, which has many immediate advantages. First, any ML model could be trained on released data without additional assumptions. Second, data from different sources could be easily pooled to build stronger models. Third, released data could be traded on data markets1, where anonymisation and protection of sensitive information is one of the biggest obstacles. Finally, data publishing would facilitate transparency and reproducibility of research studies.
In particular, we are interested in solving two problems. First, how to preserve high utility of data for ML algorithms while protecting sensitive information in the dataset. Second, how to quantify the risk of recovering private information from the published dataset, and thus, the trained model.
The main idea of our approach is to use generative
adversarial networks (GANs) (Goodfellow et al. 2014) to create
artificial datasets to be used in place of real data for
training. This method has a number of advantages over the
earlier work
value-of-data-2018/dawn-of-data-marketplace on similar public data. Third, it is more intuitive and flexible, e.g. it does not require a complex distributed architecture.
To estimate potential privacy risks, we design an ex post analysis framework for generated data. We use KL divergence estimation and Chebyshev’s inequality to find statistical bounds on expected privacy loss for a dataset in question.
Our contributions in this paper are the following: we propose a novel, yet simple, approach for private data release, and to the best of our knowledge, this is the first practical solution for complex real-world data; we introduce a new framework for statistical estimation of potential privacy loss of the released data; we show that our method achieves learning performance of model release methods and is resilient to model inversion attacks.
The rest of the paper is structured as follows. In Section 2, we give an overview of related work. Section 3 contains some preliminaries. In Section 4, we describe our approach and privacy estimation framework, and discuss its limitations. Experimental results and implementation details are presented in Section 5, and Section 6 concludes the paper. 2
In recent years, as machine learning applications become a
commonplace, a body of work on security of these methods
grows at a rapid pace. Several important vulnerabilities and
corresponding attacks on ML models have been discovered,
raising the need of devising suitable defences. Among the
attacks that compromise privacy of training data, model
inversion
Model inversion
To protect privacy while still benefiting from the use of
statistics and ML, many techniques have been developed
over the years, including k-anonymity
Most of the ML-specific literature in the area concentrates
on the task of privacy-preserving model release. One take on
the problem is to distribute training and use disjoint datasets.
For example,
A more recent line of research focuses on private data
release and providing privacy via generating synthetic
data
Similarly, our approach uses GANs, but data is generated without real seeds or applying noise to gradients. Instead, we verify experimentally that out-of-the-box GAN samples can be sufficiently different from real data, and expected privacy loss is empirically bounded by single-digit numbers. 3
This section provides necessary definitions and background. Let us commence with approximate differential privacy. Definition 1. A randomised function (mechanism) M : D ! R with domain D and range R satisfies ("; )differential privacy if for any two adjacent inputs d; d0 2 D and for any outcome o 2 R the following holds: Pr [M(d) = o] e" Pr [M(d0) = o] + : (1) Definition 2. Privacy loss of a randomised mechanism M : D ! R for inputs d; d0 2 D and outcome o 2 R takes the following form:
L(M(d)kM(d0)) = log
For more details on differential privacy and the Gaussian
mechanism, we refer the reader to
In our privacy estimation framework, we also use some classical notions from probability and information theory. Definition 4. The Kullback–Leibler (KL) divergence between two continuous probability distributions P and Q with corresponding densities p, q is given by:
DKL(P kQ) =
Z +1 1 p(x) log p(x) q(x) dx:
Note that KL divergence between the distributions of M(d) and M(d0) is nothing but the expectation of the privacy loss random variable E[L(M(d)kM(d0))].
Finally, Chebyshev’s inequality is used to obtain tail
bounds. In particular, as we expect the distribution to be
asymmetric, we use the version with semi-variances
Pr(x
E[x] + k ) k12 +22 ; where variance.
+2 = RE+[x1] p(x)(x
E[x])2dx is the upper semi4
In this section, we describe our solution, its further
improvements, and provide details of the privacy estimation
framework. We then discuss limitations of the method. More
background on privacy can be found in
The main idea of our approach is to use artificial data for
learning and publishing instead of real (see Figure 1 for a
general workflow). The intuition behind it is the following.
Since it is possible to recover training examples from ML
models
Despite the fact that the generator does not have access
to real data in the training process, one cannot guarantee
(2)
(3)
(4)
(5)
that generated samples will not repeat the input. To
alleviate this problem, we propose to enforce differential
privacy on the output of the discriminator (critic). This is done
by employing the Gaussian noise mechanism
Note that if the chosen GAN loss function was directly
differentiable w.r.t. generator output, i.e. if critic could be
treated as a black box, this modification would enforce the
same DP guarantees on generator parameters, and
consequently, all generated samples. Unfortunately, this is not the
case for practically all existing versions of GANs, including
WGAN-GP
As our evaluation shows, this modification has a number of advantages. First, it improves diversity of samples and decreases similarity with real data. Second, it allows to prolong training, and hence, obtain higher quality samples. Finally, in our experiments, it significantly improves the ability of GANs to generate samples conditionally. 4.2
Our framework builds upon ideas of empirical DP
(EDP)
As we don’t have access to exact posterior distributions, a straightforward EDP procedure in our scenario would be the following: (1) train GAN on the original dataset D; (2) remove a random sample from D; (3) re-train GAN on the updated set; (4) estimate probabilities of all outcomes and the maximum privacy loss value; (5) repeat (1)–(4) sufficiently many times to approximate ", .
If the generative model is simple, this procedure can be used without modification. Otherwise, for models like GANs, it becomes prohibitively expensive due to repetitive re-training (steps (1)–(3)). Another obstacle is estimating the maximum privacy loss value (step (4)). To overcome these two issues, we propose the following.
First, to avoid re-training, we imitate the removal of examples directly on the generated set De . We define a similarity metric sim(x; y) between two data points x and y that reflects important characteristics of data (see Section 5 for details). For every randomly selected real example i, we remove k nearest artificial neighbours to simulate absence of this example in the training set and obtain De i. Our intuition behind this operation is the following. Removing a real example would result in a lower probability density in the corresponding region of space. If this change is picked up by a GAN, which we assume is properly trained (e.g. there is no mode collapse), the density of this region in the generated examples space should also decrease. The number of neighbours k is a hyper-parameter. In our experiments, it is chosen heuristically by computing KL divergence between the real and artificial data distributions and assuming that all the difference comes from one point.
Second, we propose to relax the worst-case privacy loss
bound in step (4) by the expected-case bound, in the same
manner as on-average KL privacy. This relaxation allows us
to use a high-dimensional KL divergence estimator
Finally, after obtaining sufficiently many samples of
different pairs (De ; De i), we use Chebyshev’s inequality to
bound the probability = Pr(E[L(M(D)kM(D0))] ) of
the expected privacy loss
Our empirical privacy estimator could be improved in a number of ways. For instance, providing worst-case privacy loss bounds would be largely beneficial. Furthermore, simulating the removal of training examples currently depends on heuristics and the chosen similarity metric, which may not lead to representative samples and therefore, poor guarantees.
We provide bounds on expected privacy loss based on ex
post analysis of the artificial dataset, which is not
equivalent to the traditional formulation of DP and has certain
limitations
Lastly, all existing limitations of GANs (or generative models in general), such as training instability or mode collapse, will apply to this method. Hence, at the current state of the field, our approach may be difficult to adapt to inputs other than image data. Yet, there is still a number of privacysensitive applications, e.g. medical imaging or facial analysis, that could benefit from our technique. And as generative methods progress, new uses will be possible.
5
In this section, we describe the experimental setup and
implementation, and evaluate our method on MNIST
Learning performance experiments are set up as follows: 1. Train a generative model (teacher) on the original dataset using only the training split. 2. Generate an artificial dataset by the obtained model and use it to train ML models (students). 3. Evaluate students on a held-out test set.
Note that there is no dependency between teacher and student models. Moreover, student models are not constrained to neural networks and can be implemented as any type of machine learning algorithm.
We choose three commonly used image datasets for our experiments: MNIST, SVHN, and CelebA. MNIST is a handwritten digit recognition dataset consisting of 60000 training examples and 10000 test examples, each example is a 28x28 size greyscale image. SVHN is also a digit recognition task, with 73257 images for training and 26032 for testing. The examples are coloured 32x32 pixel images of house numbers from Google Street View. CelebA is a facial attributes dataset with 202599 images, each of which we crop to 128x128 and then downscale to 48x48. 5.2
For our experiments, we use Python and Pytorch
framework.2 We implement, with some minor modifications, a
Wasserstein GAN with gradient penalty (WGAN-GP) by
of four convolutional layers with SELU
Similarly to the original paper, we use a classical WGAN value function with the gradient penalty that enforces Lipschitz constraint on a critic. We also set the penalty parameter = 10 and the number of critic iterations ncritic = 5. Furthermore, we modify the architecture to allow for conditioning WGAN on class labels. Binarised labels are appended to the input of the generator and to the linear layer of the critic after convolutions. Therefore, the generator can be used to create labelled datasets for supervised learning.
Both networks are trained using Adam
The student network is constructed of two convolutional layers with ReLU activations, batch normalisation and max pooling, followed by two fully connected layers with ReLU, and a softmax output layer. Note that this network does not achieve state-of-the-art performance on the used datasets, but we are primarily interested in evaluating the relative performance drop compared to a non-private model.
To estimate privacy loss, we carry out the procedure
presented in Section 4. Specifically, based on recent ideas
in image qualitative evaluation, e.g. FID and Inception
Score, we compute image features by the Inception V3
network
First, we evaluate the generalisation ability of a student model trained on artificial data. More specifically, we train a student model on generated data and report test classification accuracy on a held-out real set.
As noted above, most of the work on privacy-preserving
ML focuses on model release methods and assumes
(explicitly or implicitly) access to similar ”public” data in one form
or another
We chose to compare learning performance with the
current state-of-the-art model release technique, PATE by
Table 1 shows test accuracy for the non-private baseline model (trained on the real training set), PATE, and our method. We observe that artificial data allows us to achieve 98:3% accuracy on MNIST and 87:7% accuracy on SVHN, which is comparable or better than corresponding results of PATE. These results demonstrate that our approach does not compromise learning performance, and may even improve it, while enabling the full flexibility of data release methods.
Additionally, we train a simple logistic regression model on artificial MNIST samples, and obtain 91:69% accuracy, (a) Generated (b) Real (a) Generated (b) Real compared to 92:58% on the original data, confirming that student models are not restricted to a specific type.
Furthermore, we observe that one could use artificial data for validation and hyper-parameter tuning. In our experiments, correlation coefficients between real and artificial validation losses range from 0.7197 to 0.9972 for MNIST and from 0.8047 to 0.9810 for SVHN. 5.4
Using the privacy estimation framework (see Section 4), we
fix the probability of exceeding the expected privacy loss
bound in all experiments to 10 5 and compute the
corresponding for each dataset and two versions of WGAN-GP
(vanilla and with DP critic). Table 2 summarises our
findings. It is worth noting, that our should not be viewed as
an empirical estimation of " of DP, since the former bounds
expected privacy loss while the latter–maximum. These two
quantities, however, in our experiments turn out to be similar
to deep learning DP bounds found in recent literature
The lack of theoretical privacy guarantees for our method
neccesitates assessing the strength of provided protection.
We perform this evaluation by running the model inversion
attack
In order to run the attack, we train a student model (a simple multi-layer perceptron with two hidden layers of 1000 and 300 neurons) in three settings: real data, artificial data generated by GAN (with DP critic), and real data with differential privacy (using DP-SGD with a small " < 1). As facial recognition is a more privacy-sensitive application, and provides a better visualisation of the attack, we picked CelebA attribute prediction task to run this experiment.
Figure 2 shows the results of the model inversion attack. The top row presents the real target images. The following rows depict reconstructed images from a non-private model, a model trained on GAN samples, and DP model, correspondingly. One can observe a clear information loss in reconstructed images going from non-private model, to artificial data, to DP. The latter is superior in decoupling the model and the training data, and is a preferred choice in the model release setting and/or if public data is accessible for pre-training. The non-private model, albeit trained with abundant data ( 200K images) reveals facial features, such as skin and hair colour, expression, etc. Our method, despite failing to conceal general shapes in training images (i.e. faces), seems to achieve a trade-off, hiding most of the specific features. The obtained reconstructions are either very noisy (columns 1, 2, 6, 8), much like DP, or converge to some average feature-less faces (columns 4, 5, 7).
We also analyse real and reconstructed image pairs using
OpenFace
To evaluate our privacy estimation method, we look at how the privacy loss bound correlates with the success of the attack. Figure 3 depicts the privacy-accuracy trade-off curve for an MLP (64-32-10) trained on artificial data. In this setting, we use a stacked denoising autoencoder to compress images to 64-dimensional feature vectors and facilitate the attack performance. Along the curve, we plot examples of the model inversion reconstruction at corresponding points. We see that with growing , meaning lower privacy, both model accuracy and reconstruction quality increase.
Finally, as an additional measure, we perform visual inspection of generated examples and corresponding nearest neighbours in real data. Figures 4 and 5 depict generated and the corresponding most similar real images from SVHN and CelebA datasets. We observe that, despite general visual similarity, generated images differ from real examples in details, which is normally more important for privacy. For SVHN, digits vary either in shape, colour or surroundings. A lot of pairs come from different classes. For CelebA, the pose and lighting may be similar, but such details as gender, skin colour, facial features are usually significantly different.
We investigate the problem of private data release for complex high-dimensional data. In contrast to commonly studied model release setting, this approach enables important advantages and applications, such as data pooling from multiple sources, simpler development process, and data trading.
We employ generative adversarial networks to produce artificial privacy-preserving datasets. The choice of GANs as a generative model ensures scalability and makes the technique suitable for real-world data with complex structure. Unlike many prior approaches, our method does not assume access to similar publicly available data. In our experiments, we show that student models trained on artificial data can achieve high accuracy on MNIST and SVHN datasets. Moreover, models can also be validated on artificial data.
We propose a novel technique for estimating privacy of released data by empirical bounds on expected privacy loss. We compute privacy bounds for samples from WGAN-GP on MNIST, SVHN, and CelebA, and demonstrate that expected privacy loss is bounded by single-digit values. To evaluate provided protection, we run a model inversion attack and show that training with GAN reduces information leakage (e.g. face detection drops from 63:6% to 1:3%) and that attack success correlates with estimated privacy bounds.
Additionally, we introduce a simple modification to the critic: differential privacy layer. Not only does it improve privacy loss bounds and ensures DP guarantees for the critic output, but it also acts as a regulariser, improving stability of training, and quality and diversity of generated images.
Considering the rising importance of privacy research and the lack of good solutions for private data publishing, there is a lot of potential future work. In particular, a major direction of advancing current work would be achieving differential privacy guarantees for generative models while still preserving high utility of generated data. A step in another direction would be to improve the privacy estimation framework, e.g. by bounding maximum privacy loss, or finding a more principled way of sampling from outcome distributions.