Privacy policies are documents that describe what data is collected by a website or an app and how that data is handled. Privacy policies are often long and difficult to understand. Recently people have started to turn to Natural Language Processing (NLP) to automatically extract statements from the text of these policies. This article reports on a study to evaluate the benefits of using word embeddings in this endeavor. Specifically, we use 150,000 privacy policies to build word vectors in an unsupervised manner. This includes evaluating the benefits of privacy specific word embeddings. Evaluation is conducted on the OPP-115 corpus of privacy policy annotations. By building privacy-specific embeddings we hope to accelerate research at the intersection of privacy policies and language technologies.
Privacy policies tend to be long and complex documents that
are often difficult to understand. They are the primary
mechanism to inform users about the collection and handling of
their data. Over the past several years, there has been a
growing interest in automating the understanding of privacy
policies using Machine Learning and Natural Language
Processing techniques (e.g.,
The contributions of our paper can be summarized as follows. 1. We investigate the utility of in-domain word embeddings, and find that they indeed help over generic word embeddings to obtain better segment-labeling performance in the privacy domain. We observe meaningful improvements on the OPP-115 corpus. We measure an average macro F1 of 0.803 on the test set. 2. We empirically investigate the relationship between dimensionality of the word embeddings and segment labeling performance. We look at the performance across a diverse set of important data-practice categories when the dimensionality of the word embeddings changes. 3. We investigate the number of policies that are required to train expressive word embeddings. We have access to over 300,000 privacy policies which we scraped from the Google Play Store. We investigate the amount of policies which are needed to get good results on the OPP-115 dataset. Henceforth, we call this the policy corpus. It is to be noted that we did not need all 300,000 privacy policies as the performance saturates fairly quickly. Incidentally, we would have needed significantly more computational resources to train word embeddings on all 300,000 policies. This is not to say that word embeddings trained on all 300,000 policies might not have helped in the context of different, possibly more subtle data practices than those considered in the OPP-115 dataset. 4. We present a qualitative analysis of representations of words in vector space in the privacy domain. We want to understand how in-domain word embeddings differ from the generic word embeddings which are domain independent.
2
Word embeddings have a rich history in the Natural
Language Processing community, and have proven to be
useful in a wide variety of tasks. Work on neural models for
formulating word representations was reported as early as
2003 by
3
Fast text was introduced by
We store our learned embeddings for future access. We created separate word embedding stores for the different kinds of embeddings we built. We experimented with the dimensionality and the amount of data required to train good word embeddings. These results are discussed in the ”Results and discussions” section.
We have to convert the text representation of the OPP-115
corpus to real valued representation so that our neural
models can start using them. But, if we have a large vocabulary it
becomes difficult for networks to fit the data. So we restrict
our vocabulary size to the top 50,000 words in the
vocabulary. We also preprocess the data to convert all the words
to lower case. We use all the pre-processing techniques used
by
We try several neural architecture models. Following
OPP-115
It is a well documented fact that words which appear in
similar contexts must be close to each other in the vector space
4
The OPP-115 data-set is a multi-label classification
problem which has skewed counts on some of the classes. For a
detailed understanding of these privacy policies we ask the
reader to refer to:
Some of the classes in this dataset have fairly low counts
of positive examples, making it difficult for machine
learning models to learn the classification. For instance, in
previous publications, the authors had found it hard to identify
segments associated with the ”Data retention” class, as this
class only has a few positive instances in the corpus
We used our word embeddings, averaged them and used that as the input to the logistic regression model. The results of the model are shown in Table 1.
We see that our baseline model has some improvement
in performance over the results reported by
Feed-forward neural networks are powerful machine learning models which perform really well on classification tasks. We used a feed-forward neural network as shown in Figure 3. The input layer to this network was the averaged embeddings of all the words in the sentence.
More formally, given a list of words as a segment, we perform the following: wordVectors = getW ordV ectors(w1; w2; w3:::wn) averageEmbeddings = Pi2n vectori / n
Here w1; w2:::wn are the words of a sentence. We convert these text of words into their vector representation to get vectori . We then average these vectors to get averageEmbeddings .
These average embeddings were then used as the input
layer to a two layer network. Each layer of the network
had 64 Rectified Linear Unit (Relu)
This approach of text classification is generally
considered to be a strong baseline by the NLP community (e.g.,
Our CBOW Model gave us the best results. We report our
accuracies in Table 2. It can be seen that our results are the
state of the art on this dataset.
We compare our results with the F1 score of
We get a macro F1 of 0.803 when compared to their macro
F1 of 0.667 on the classes which we have chosen to
evaluate our model. We also compare our results with Liu et
al. (2016a). We see that we get better results than
We also tried deep convolutional models by convolving over the word embeddings. We used two sets of convolutional layers , with 200 filters each, but with strides of 3 and 5. We used Relu non-linearity and Maxpooling operation before being fed to the next set of these CONV-RELU-MaxPool Layers. The detailed architecture for our model is shown in Figure 4.
Although this model generally does better over the
baselines described by
The results of our convolutional model is shown in Table 3. It can be seen that our model is very close to the CBOW model, but does not outperform it as the number of training examples is fairly low in the OPP-115 corpus. The simple feed forward network is able to capture the relationship better as the number of parameters in this model is orders of magnitude lower than in the deeper model. This is consistent with the general observation that as the model parameters increase, one needs more data to get better results.
We also explored the use of contextualized word
embeddings, using BERT
From Table 4 we see that we get good results on classes that have a higher number of positive instances. BERT does not do as well on classes with a lower number of positive instances. This is consistent with the observations across a lot of NLP tasks such as Question Answering or Natural Language Inference, where authors have reported significantly better results when BERT was used for classification tasks. It is to be noted that we don’t do any hyper-parameter tuning with BERT, we just take an off-the-shelf model and train on the OPP-115 corpus for 3 epochs.
7
In this section we try and answer research questions that are at the intersection of privacy policies and language technologies.
In this section we compare our non contextualized word
embeddings to the non contextualized generic word
embeddings. We believe it is not fair to compare our word
embeddings to BERT, because BERT is not an embedding, but
more of a language model to represent sentences. Here we
focus on evaluating the benefits of non-contextualized
indomain word embeddings over non-contextualized generic
word embeddings. We use GloVe embeddings
It is also worth noting that the standard deviation of F1 scores is higher for practices with lower numbers of positive examples. In this paper, we have report average F1 scores across the different validation folds, along with their standard deviations. This is to capture the variability of the F1 score, as one might observe when looking at unseen data.
We checked the performance of both the GloVe and our indomain embeddings with different settings of the embeddings size (100 and 300). As can be seen, the 300 dimensional embeddings performed better than the 100 dimensional embeddings. The results of this experiment is provided in Table 7.
From Table 7, it can be observed that the higher dimensional embeddings tend to give better results over their lower dimensional counterparts. The higher order dimension takes longer to train, but since it has more dimensions it can capture relationships between words in a more expressive fashion.
Performance for different values of the embeddings are shown in from Table 7. Results presented in earlier tables are for 300 dimensional embeddings, as these embeddings performed than the 100 dimensional ones.
We now turn our attention to trying to answer the question of how many privacy policies are needed to get good embeddings for the OPP-115 dataset classification task. For this, we trained 100 dimensional FastText embeddings using different numbers of privacy policies and looked at the F1 score across various categories.
We observe from figure 7 that there is a plateau in the average F1 score which can be obtained after 20,000 policies. It is also interesting to note that classes which have higher positive examples in this dataset, tend to plateau quicker than the ones which have lower numbers of positive examples. For example, the “First Party Collection and Use” class has a lot more positive examples than the “Data Retention” class. We can see in figure 7 that the “Data Retention” class also takes a lot more time than the‘First Party Collection and Use” to reach the maximum F1 observed using our model.
After training word embeddings we would want to visualize a how the embeddings look like in the high dimensional vector space. For this, we take the 300 dimensional embeddings which we trained using our policy corpus and project them onto a 2D space using Principal Component Analysis (PCA). We compare the result of this projection with GloVe to see if our domain-specific model captures domain semantics better than GloVe.
It can be observed in Figure 7 that the privacy related terms are very closely clustered. In Figure 5 we see that there is no special semantics for privacy related words. The terms “food” and “party” are closely related, as that is the more common case outside of the domain of privacy policies. But in the Figure 7 “party” is more closely associated with “privacy” because of the privacy domain-specific concept of “third-party collection.” The terms “cookies,” “track,” and “browser” are three words that commonly appear together in privacy policies. We observe that these words are closely related in our in-domain vector space. It can also be observed that the terms ”data” and ”collection” are closer to each other in our in-domain vector space when compared to the more general vector space of GloVe. This is because privacy policies talk about users’ data collection most of the time.
While anecdotal, these observations suggest that the improvement in performance resulting from the use of indomain embeddings can be attributed to these embeddings being able to better capture unique syntactic and semantic features of privacy policies.
8
In this paper we presented distributed word vector representations for privacy policies. We showed that in-domain embeddings can yield performance improvements on privacy policy related tasks over the use of generic embeddings such as GloVe. We reported good F1 scores across different data practice categories using our domain specific embeddings. We also showed both quantitatively and qualitatively that our in-domain word embeddings help improve performance of privacy policy segment labeling tasks on the OPP-115 corpus of privacy policies.
This study was supported in part by the NSF Frontier grant on Usable Privacy Policies (CNS-1330596). The US Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF or the US Government. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing high performance computing resources that have contributed to the research results reported within this paper.