=Paper=
{{Paper
|id=Vol-2335/paper4
|storemode=property
|title=Natural Language Processing for Mobile App Privacy Compliance
|pdfUrl=https://ceur-ws.org/Vol-2335/1st_PAL_paper_6.pdf
|volume=Vol-2335
|authors=Peter Story,Sebastian Zimmeck,Abhilasha Ravichander,Daniel Smullen,Ziqi Wang,Joel Reidenberg,N. Cameron Russell,Norman Sadeh
}}
==Natural Language Processing for Mobile App Privacy Compliance==
https://ceur-ws.org/Vol-2335/1st_PAL_paper_6.pdf
Natural Language Processing for Mobile App Privacy Compliance
Peter Story∗1 , Sebastian Zimmeck∗2 , Abhilasha Ravichander1 , Daniel Smullen1 ,
Ziqi Wang1 , Joel Reidenberg3 , N. Cameron Russell3 , and Norman Sadeh∗1
1
School of Computer Science, Carnegie Mellon University
2
Department of Mathematics and Computer Science, Wesleyan University
3
School of Law, Fordham University
Abstract to the service’s practices, they are not allowed to use the
service. Natural language privacy policies are intended to
Many Internet services collect a flurry of data from their notify users of privacy practices. Privacy policies are com-
users. Privacy policies are intended to describe the ser-
vices’ privacy practices. However, due to their length
plex and lengthy documents: they are often vague, internally
and complexity, reading privacy policies is a challenge contradictory, offer little protection, or are silent on critical
for end users, government regulators, and companies. points (Marotta-Wurgler 2015). While there are other forms
Natural language processing holds the promise of help- of privacy notification, such as mobile app permission re-
ing address this challenge. Specifically, we focus on quests, these are not a replacement for privacy policies; per-
comparing the practices described in privacy policies to mission requests are generally insufficient to express what
the practices performed by smartphone apps covered by users agree to with sufficient clarity. Machine-readable pri-
those policies. Government regulators are interested in vacy policies, such as P3P policies (Cranor et al. 2002),
comparing apps to their privacy policies in order to de- were suggested as replacements for natural language privacy
tect non-compliance with laws, and companies are in- policies. However, none of these replacements have gained
terested for the same reason.
widespread adoption. Thus, despite their shortcomings, nat-
We frame the identification of privacy practice state-
ments in privacy policies as a classification problem,
ural language privacy policies are the standard instrument
which we address with a three-tiered approach: a pri- for effectuating notice and choice.
vacy practice statement is classified based on a data The FTC engages in enforcement actions against opera-
type (e.g., location), party (i.e., first or third party), and tors of apps that are non-compliant with their privacy poli-
modality (i.e., whether a practice is explicitly described cies. Such non-compliance is considered an unfair or de-
as being performed or not performed). Privacy policies ceptive act or practice in or affecting commerce in viola-
omit discussion of many practices. With negative F1 tion of Section 5(a) of the FTC Act (FTC 2014). In order
scores ranging from 78% to 100%, the performance re- to detect whether an app is potentially not compliant with
sults of this three-tiered classification methodology sug- its privacy policy, we built the Mobile App Privacy System
gests an improvement over the state-of-the-art. (MAPS) (Zimmeck et al. 2019). MAPS is of interest to both
Our NLP analysis of privacy policies is an integral part government regulators and companies. For government reg-
of our Mobile App Privacy System (MAPS), which
we used to analyze 1,035,853 free apps on the Google
ulators, MAPS can identify potential compliance issues, re-
Play Store. Potential compliance issues appeared to be ducing the cost of investigations. For companies, MAPS can
widespread, and those involving third parties were par- help them ensure that their privacy policies fully describe
ticularly common. their apps’ practices.
Our focus in this article is on the natural language analy-
sis component of MAPS. We provide a detailed description
1 Introduction of the design and performance of our three-tiered classifier
In the absence of a general privacy law in the United States, design for identifying privacy practice statements (§ 3). We
the Federal Trade Commission (FTC) is stepping into the also provide a summary of findings from our recent scan of
void and is creating a “common law of privacy” (Solove and over 1 million mobile apps on the Google Play Store (§ 4).
Hartzog 2014), which, to a large extent, is based on the no- In this large scale analysis of policies and apps, we found
tice and choice paradigm. In this paradigm, users are noti- widespread evidence of potential privacy compliance issues.
fied of a service’s privacy practices and are given a choice In particular, it appears that many apps’ privacy policies do
to consent to those practices; if the user does not consent not sufficiently disclose identifier and location data access
∗
practices performed by ad networks and other third parties.
The first two authors contributed equally to this study. Previ-
ously, Sebastian Zimmeck was a postdoctoral associate at Carnegie
Mellon’s School of Computer Science. The corresponding authors’
2 Related Work
emails are: pstory@andrew.cmu.edu, szimmeck@wesleyan.edu, Our work leverages earlier studies on the automated analysis
and sadeh@cs.cmu.edu. of privacy policy text as well as examinations of privacy in
the Android ecosystem. privacy practices. Story et al. studied the metadata of over a
million apps on the Play Store and found that many apps lack
Automated Privacy Policy Text Analysis privacy policies, even when developers describe their apps
Privacy policies are the main instruments for disclosing and as collecting users’ information (Story, Zimmeck, and Sadeh
describing apps’ or other software’s privacy practices. How- 2018). Analyzing close to a million Android web apps (i.e.,
ever, the sheer volume of text an individual user would need Android apps that use a WebView), Mutchler et al. found
to read for the software he or she is using makes privacy poli- that 28% of those have at least one vulnerability, such as
cies impractical for meaningfully conveying privacy prac- data leakage through overridden URL loads (Mutchler et al.
tices (McDonald and Cranor 2008). Some research has fo- 2015). Differences in how apps are treating sensitive data
cused on the structure of privacy policies. For example, the were used to identify malicious apps (Avdiienko et al. 2015).
problem of identifying policy sections relating to different More recently, AppCensus revealed that many Android apps
topics (Ramanath et al. 2014; Liu et al. 2018). Sathyendra collect persistent device identifiers to track users, which
et al. classified advertising opt outs and similar consumer is not allowed for advertising purposes according to the
choice options on websites (Sathyendra et al. 2017). Other Google Play Developer Program Policy (Reyes et al. 2018;
work has focused on building tools for users. Using a sim- Google 2018b). The observation of 512 popular Android
ple naive Bayes classifier, Zimmeck and Bellovin provided apps over eight years of version history by Ren et al. came
a browser extension for identifying common privacy prac- to the conclusion that an important factor for higher privacy
tices in policy text (Zimmeck and Bellovin 2014). Tesfay et risks over time is the increased number of third party do-
al. used a machine learning-based approach to identify text mains receiving personally identifiable information (Ren et
addressing various GDPR provisions (Tesfay et al. 2018). al. 2018). In line with these observations, it is one of our
Harkous et al. developed PriBot, a chatbot for answering goals in this study to examine apps’ third party practices in
questions about privacy policies (Harkous et al. 2018). Dif- the Android ecosystem.
ferent from those studies, however, our domain consists of
app policies instead of website policies. 3 Analysis Techniques
Various studies analyzed privacy policies in specific do-
mains. Cranor et al. evaluated financial institutions’ privacy Our Mobile App Privacy System (MAPS) is comprised of
notices, which, in the US, nominally adhere to a model pri- separate modules for the analysis of privacy policies and
vacy form released by federal agencies (Cranor, Leon, and apps. Our system compares the policy and app analyses in
Ur 2016). They found clusters of institutions sharing con- order to identify potential compliance issues.
sumer data more often than others. They also found institu-
tions that do not follow the law, by disallowing consumers Privacy Practices
to limit such sharing. Further, Zhuang et al. aimed to help
Our system analyzes privacy practices. A privacy practice,
university researchers by automating enforcement of privacy
or simply practice, describes a behavior of an app that
policies of Institutional Review Boards (Zhuang et al. 2018).
can have privacy implications. Table 3 contains the list of
Auditing the disclosure of third party data collection prac-
practices we consider in our model.1 We account for the
tices on 200,000 website privacy policies, Libert found that
fact that disclosures found in privacy policies can vary in
the names of third parties are usually not explicitly disclosed
specificity. For instance, for the access of location data our
in website privacy policies (Libert 2018). We focus on clas-
model includes practices that pertain to location in general
sifying first and third party access of contact, location, and
(i.e., Location) as well as more specific practices that ex-
unique identifier data in smartphone apps’ privacy policies.
plicitly identify the type of access (i.e., Location Cell
Android Privacy Studies Tower, Location GPS, and Location WiFi). Our
model distinguishes between first party access, where data
We are extending the emerging domain of verifying pri- is accessed by the code of the app itself, and third party ac-
vacy practices of mobile apps against privacy requirements, cess, where data is accessed by advertising or other third
notably privacy policies. The closest related work to ours party libraries. Finally, our model also distinguishes between
analyzed the practices of 17,991 Android apps and de- a policy describing the performance of a practice (e.g., “We
termined whether those with a privacy policy adhered to access your location information.”) and the description that
it (Zimmeck et al. 2017). Several other studies have also a practice is not performed (e.g., “We do not access your
compared privacy policies to apps’ code (Yu et al. 2016; location information.”). When access is neither explicitly
Slavin et al. 2016). Going beyond this work, our system is described nor explicitly denied, neither modality classifier
capable of large-scale analyses, which we demonstrate by flags the statement. Note that a given text fragment can refer
an analysis of 1,035,853 free apps on the Google Play Store. to multiple practices.
Additionally, our analysis evaluates compliance issues at a
finer granularity. This advance is notable because the access 1
In preliminary tests we also considered city, ZIP code, postal
of coarse-grained location data (e.g., city) is far less privacy- address, username, password, ad ID, address book, Bluetooth, IP
invasive than the access of fine-grained data (e.g., latitude address (identifier and location), age, and gender practices. How-
and longitude). ever, we ultimately decided against further pursuing those as we
We are motivated to study privacy in the Android ecosys- had insufficient data, unreliable annotations, or difficulty identify-
tem due to numerous findings of potentially non-compliant ing a corresponding API for the app analysis.
Privacy Policy Analysis
We characterize the detection of privacy practice descrip-
tions in privacy policy text as a classification problem.
Dataset We used the APP-350 corpus of 350 annotated
mobile app privacy policies to train and test our classi-
fiers (Zimmeck et al. 2019).2 The corpus’s policies were se-
lected from the most popular apps on the Google Play Store.
The policies were annotated by legal experts using a set of
privacy practice annotation labels. As they were annotating
the policies, the experts also identified the policy text frag-
ments corresponding to the practice annotation labels they
applied. All policies were comprehensively annotated. Con-
sequently, it is assumed that all unannotated portions of text
do not describe any of the practices and can be used as train-
ing, validation, and test data to detect the absence of state- Figure 1: By decomposing the classification task into three
ments on respective practices. subproblems more positive training samples are available
We randomly split the annotated privacy policies into than for monolithic classifiers.
training (n = 188), validation (n = 62), and test (n = 100)
sets. We used the training and validation sets to develop our
classifiers. The test set was set aside in order to prevent over- and 2 “modality” classifiers). These classifiers have a much
fitting. We did not calculate performance using the test set higher number of positive samples available for training, as
until after we finished developing our classifiers. shown in Figure 1.
Preprocessing As classifier performance depends on ade-
Classification Task The goal of the classification task is
quate preprocessing of policy text as well as domain-specific
to assign annotation labels to policy segments, that is, struc-
feature engineering, we normalize whitespace and punctua-
turally related parts of policy text that loosely correspond to
tion, remove non-ASCII characters, and lowercase all pol-
paragraphs (Wilson et al. 2016; Liu et al. 2018). We focus
icy text. Because stemming did not lead to performance im-
on segments instead of entire policies to make effective use
provements, we are omitting it. In order to run our classifiers
of the annotated data and to identify the specific policy text
on the most relevant set of features, we use an optional pre-
locations that describe a certain practice.
processing step of sentence filtering. Based on a grid search,
The infrequent occurrence of certain types of statements
in cases where it improves classifier performance, we re-
makes the training of classifiers for some practices more
move a segment’s sentences from further processing if they
challenging. In particular, statements on third party prac-
do not contain keywords related to the classifier in ques-
tices and statements explicitly denying that activities are
tion (Zimmeck et al. 2017). For example, the Location
performed are rare. For example, our training set only in-
classifier is not trained on sentences which only describe
cludes 7 segments saying that Location Cell Tower
cookies.
information is not accessed by third parties. To address this
challenge, we decompose the classification problem into Vectorizing Prior to training, we generate vector repre-
three subproblems, that is, classifying (1) data types (e.g., sentations of the segments. Specifically, we take the union
Location), (2) parties (i.e., 1stParty or 3rdParty)3 , of a TF-IDF vector and a vector of manually crafted fea-
and (3) modalities (i.e., whether a practice is explicitly tures. Our TF-IDF vector is created using the TfidfVector-
described as being performed or not performed). For ex- izer (scikit-learn developers 2016a) configured with English
ample, the Location Cell Tower 3rdParty Not stopwords (stop words=’english’), unigrams and bi-
Performed classification will be assigned to a segment grams (ngram range=(1, 2)), and binary term counts
if the Location Cell Tower, 3rdParty, and Not (binary=True). This configuration is similar to what was
Performed classifiers all return a positive result for the used in prior work (Liu et al. 2018). Our vector of manu-
segment. ally crafted features consists of Boolean values indicating
The decomposition of the classification task allows for the presence or absence of indicative strings we observed in
an economic use of annotated data. If the subproblems our training and validation data. For example, we include the
were tackled all at once, 68 monolithic classifiers would be string not collect, because we realized that it would be
needed, most of which would have to be trained on fewer a strong indicator of the negative modality.
than 100 positive training samples. By dividing the problem,
only 22 classifiers are needed (18 “data type”, 2 “party”, Training Using scikit-learn, version 0.18.1 (Pedregosa
et al. 2011) we train binary classifiers for each data
2
The dataset is available at https://data.usableprivacy.org. type, party, and modality. For all but four classifiers
3
Note that the Single Sign On and Single Sign On: we use scikit-learn’s SVC implementation (scikit-learn
Facebook practices do not use a party classifier, as all data is developers 2016b). We train those with a linear ker-
exchanged between the app developer as first party and the SSO nel (kernel=’linear’), balanced class weights
provider as third party. (class weight=’balanced’), and a grid search with
Configuration
Classifier Baseline + Bigrams + C.F. + S.F. + Final Performance Analysis Table 3 shows the performance of
Contact 29% +42% +35% +28% +39% the classifiers on the privacy policies of the test set. We say a
Contact Email Address 73% +9% +8% +12% +11% policy describes a practice if at least one segment is flagged
Contact Phone Number 76% +11% +2% +11% +9%
Identifier Cookie 88% +2% +2% +3% +2% by the corresponding data type, party, and positive modal-
Identifier Device ID 74% +9% +8% +12% +13% ity classifiers. Since our definition of potential compliance
Identifier IMEI 77% -19% +20% +17% +17%
Identifier MAC 84% -23% +2% -2% -2%
issues does not depend on the negative modality classifier,
Identifier Mobile Carrier 62% -14% -3% 0% -14% we do not include it in the table. Because detecting potential
Location 80% +4% +3% +9% +6% compliance issues is dependent on detecting when practices
Location Cell Tower 62% -4% +4% +12% +10%
Location GPS 76% -1% +3% +16% +12% are not described in policies (Zimmeck et al. 2017), nega-
Location WiFi 74% +5% +5% +1% +5% tive predictive value, specificity, and negative F1 scores are
Single Sign On 63% +7% +11% -43% +4% of particular importance.
Single Sign On: Facebook 75% +3% +5% -64% +6%
1stParty 95% +0% -1% -1% +0% In the closest related work (Zimmeck et al. 2017), classi-
3rdParty 77% +4% +2% +1% +1% fiers for contact, identifier, and location data practices cov-
Performed 90% +1% +5% -2% +6%
Not Performed 73% -2% +13% +5% +14%
ered multiple specific practices. Thus, a direct performance
comparison to our classifiers is not possible. However, with
Table 1: Effects of different preprocessing and feature con- negative F1 scores ranging from 78% to 100%, 23 of our
figurations on our classifiers’ F1 scores. Effects are calcu- specific classifiers achieve better negative F1 scores than the
lated with regard to a baseline configuration (Baseline), in corresponding course-grained classifiers, and 3 performed
which the TF-IDF vectors only include unigrams. For the equally. These results demonstrate that our approach consti-
baseline, bigrams (Bigrams) and manually crafted features tutes an overall improvement over the state of the art. We
(C.F.) are not included, and keyword-based sentence filter- believe that decomposing the classification task into three
ing (S.F.) is not performed. For example, including bigrams subproblems increases performance as it allows for a better
in our TF-IDF vectors leads to an F1 score increase of 42% exploitation of training data compared to monolithic classi-
(from 29% to 71%) for the Contact classifier. Our final con- fiers.
figuration (Final) includes bigrams as well as crafted fea- Our results reveal that generally + support is lower for
tures; sentence filtering is enabled on a per-classifier basis third party practices; that is, third party practices are often
using a grid search. not as extensively described in privacy policies as first party
practices. It should be further noted that higher counts of -
Configuration Improved Avg. Improvement Decreased Avg. Decrease support generally correlate with higher performance scores.
+ Bigrams 11/18 +8.8% 6/18 -10.5% Intuitively, it is easier to classify a policy that does not de-
+ Crafted Features 16/18 +8.0% 2/18 -2.0%
+ Sentence Filtering 12/18 +10.6% 5/18 -22.4%
scribe a practice, which makes up the majority of - support
Final 15/18 +10.3% 2/18 -8.0% instances.
We reviewed the errors made by our classifiers and iden-
Table 2: Summary of effects of different preprocessing and tified several potential areas for improvement. First, ap-
feature configurations on F1 scores based on the data shown proaching the classification task at the level of segments,
in Table 1. For example, adding bigrams (+ Bigrams) to our as suggested by prior work (Wilson et al. 2016; Liu et al.
TF-IDF vectors improved the F1 scores of 11 classifiers by 2018), can pose difficulties for our subproblem classifiers.
an average of 8.8%, but also decreased the F1 scores for 6 For example, if a segment describes a 1stParty perform-
classifiers by an average of 10.5%. ing the Location practice, and a 3rdParty performing
Contact, our classifiers cannot distinguish which party
should be associated with which practice. Thus, perform-
five-fold cross-validation over the penalty (C=[0.1, 1, ing classifications at the level of sentences may yield per-
10]) and gamma (gamma=[0.001, 0.01, 0.1]) pa- formance improvements. Second, the variety of technical
rameters. We create rule-based classifiers for four data types language in privacy policies poses challenges. For exam-
(Identifier, Identifier IMSI, Identifier ple, we observed a false positive when “location” was used
SIM Serial, and Identifier SSID BSSID) due to in the context of “co-location facility”, and a false negative
the limited amount of data and their superior performance. when “clear gifs” was used to refer to web beacons. Such
Our rule-based classifiers identify the presence or absence errors might be prevented by training on more data or using
of a data type based on indicative text strings. domain-specific word embeddings (Kumar et al. 2019). Fi-
Table 1 shows the effects of our features and preprocess- nally, a more sophisticated semantic representation might be
ing steps on the F1 scores of our non-rule-based classifiers. necessary in certain cases. For example, we observed mis-
The performance is calculated using our training and valida- classification of a sentence which said that although the first
tion sets. We made sentence filtering an optional part of pre- party does not perform a practice, third parties do perform
processing because of the large detrimental effect it has on the practice.
some of our classifiers, as highlighted in Table 2. In general,
our results suggest that the chosen feature and preprocess- App Analysis
ing steps improve classifier performance. However, ideally MAPS detects apps’ privacy practices at app store-wide
they should be chosen on a per-classifier basis to avoid any scale. Detecting which practices an app performs relies on
negative performance impact. static code analysis, a relatively resource-efficient technique
Policy Classification NPV Specificity Neg. F1 Precision Recall F1 +/- Support
Contact 1stParty 92% 96% 94% 89% 80% 84% 30/70
Contact 3rdParty 95% 96% 95% 43% 38% 40% 8/92
Contact Email Address 1stParty 78% 90% 84% 97% 94% 96% 80/20
Contact Email Address 3rdParty 91% 83% 87% 29% 46% 35% 13/87
Contact Phone Number 1stParty 93% 93% 93% 94% 94% 94% 54/46
Contact Phone Number 3rdParty 97% 93% 95% 22% 40% 29% 5/95
Identifier 1stParty 93% 68% 78% 38% 80% 52% 20/80
Identifier 3rdParty 97% 76% 85% 21% 75% 33% 8/92
Identifier Cookie 1stParty 100% 92% 96% 95% 100% 98% 63/37
Identifier Cookie 3rdParty 94% 92% 93% 92% 94% 93% 52/48
Identifier Device ID 1stParty 86% 96% 91% 96% 87% 91% 54/46
Identifier Device ID 3rdParty 97% 95% 96% 83% 90% 86% 21/79
Identifier IMEI 1stParty 99% 99% 99% 94% 94% 94% 17/83
Identifier IMEI 3rdParty 99% 100% 99% 100% 75% 86% 4/96
Identifier IMSI 1stParty 100% 100% 100% 100% 100% 100% 3/97
Identifier IMSI 3rdParty 99% 100% 99% N/A 0% 0% 1/99
Identifier MAC 1stParty 95% 98% 96% 88% 79% 83% 19/81
Identifier MAC 3rdParty 99% 96% 97% 56% 83% 67% 6/94
Identifier Mobile Carrier 1stParty 90% 100% 95% 100% 57% 73% 21/79
Identifier Mobile Carrier 3rdParty 98% 97% 97% 25% 33% 29% 3/97
Identifier SIM Serial 1stParty 100% 97% 98% 73% 100% 84% 8/92
Identifier SIM Serial 3rdParty 100% 99% 99% 50% 100% 67% 1/99
Identifier SSID BSSID 1stParty 99% 100% 99% 100% 80% 89% 5/95
Identifier SSID BSSID 3rdParty 100% 99% 99% N/A N/A N/A 0/100
Location 1stParty 92% 81% 86% 87% 95% 91% 58/42
Location 3rdParty 96% 83% 89% 61% 87% 71% 23/77
Location Cell Tower 1stParty 98% 94% 96% 71% 86% 77% 14/86
Location Cell Tower 3rdParty 98% 95% 96% 29% 50% 36% 4/96
Location GPS 1stParty 99% 94% 96% 88% 97% 92% 29/71
Location GPS 3rdParty 99% 94% 96% 45% 83% 59% 6/94
Location WiFi 1stParty 99% 86% 92% 48% 92% 63% 12/88
Location WiFi 3rdParty 100% 95% 97% 29% 100% 44% 2/98
Single Sign On 89% 90% 90% 83% 81% 82% 37/63
Single Sign On: Facebook 95% 84% 89% 72% 91% 81% 32/68
Table 3: Performance metrics of our classifiers for determining whether or not a privacy policy states that a practice is performed
calculated on the test set (n = 100). The negative predictive value (NPV) is the precision for negative instances. Specificity is
the recall for negative instances. Negative F1 (Neg. F1) is the F1 for negative instances. In the Support column, + is the number
of ground-truth positive instances (cases where policies truly describe the practice being performed) and - is the number of
ground-truth negative instances (cases where policies truly do not describe the practice being performed). With negative F1
scores ranging from 78% to 100%, the absence of all practices can be classified relatively accurately. Lower positive F1 scores,
for example, 40% for access of contact information by third parties, could be the result of insufficient availability of data. N/A
is shown where the metrics are undefined, or where a lack of positive ground truth instances would always make the metric
zero.
compared to dynamic code analysis. Our system operates ual dynamic analysis.
on four app resources: Android APIs, permissions, strings,
and class structure. If a sensitive Android API is called, the Compliance Analysis
app has the required permissions to make the call, and re-
quired string parameters (e.g., the GPS PROVIDER string) Our system combines policy and app analysis results to iden-
are passed in, the system will flag the existence of a first tify potential compliance issues. We define a potential com-
or third party practice depending on the package name of pliance issue to mean that an app is performing a practice
the class from which the call originated. We assume a threat (e.g., Location GPS 1stParty) while its associated
model which considers data as compromised from the mo- policy does not disclose it either generally (e.g., “Our app
ment a privacy-sensitive API appears to be called (Neisse et accesses your location data.”) or specifically (e.g., “Our app
al. 2016). accesses your GPS data.”). We chose this definition because
After downloading an app from the Google Play Store our we observed that policies generally either disclose that a
system decompiles it into Smali bytecode using Apktool.4 practice is performed or omit discussion of the practice—
It then searches through the bytecode, identifying APIs in- statements denying practices are rare.
dicative of a privacy practice being performed. Generally, if Table 4 shows our system’s identification of potential
a practice occurs in a package corresponding to the app’s compliance issues and its performance. For the 26 practices
package ID, the practice is considered a first party practice; for which positive ground truth instances were present, we
otherwise, it is considered a third party practice. In order to observe a mean F1 score of 71%. Many potential compliance
evaluate the performance of our system’s app analysis, we issues relate to the access of identifiers. However, the three
compare its results against ground truth obtained by a man- third party location practices Cell Tower, GPS, and WiFi ac-
count for 15, 10, and 12 respective findings as well. Notably,
4 all first party practices exhibit a lower number of potential
Apktool, https://ibotpeaches.github.io/Apktool/, accessed:
March 18, 2019. compliance issues than their third party counterparts.
Potential Compliance Issue Precision Recall F1 +/-/? Support
Contact Email Address 1stParty 75% 75% 75% 4/77/19
Contact Email Address 3rdParty 38% 71% 50% 7/74/19
Contact Phone Number 1stParty 100% 100% 100% 1/82/17
Contact Phone Number 3rdParty 29% 67% 40% 3/80/17
Identifier Cookie 1stParty 50% 100% 67% 1/70/29
Identifier Cookie 3rdParty 83% 87% 85% 23/48/29
Identifier Device ID 1stParty 70% 88% 78% 16/63/21
Identifier Device ID 3rdParty 96% 86% 91% 58/21/21
Identifier IMEI 1stParty 79% 65% 71% 17/64/19
Identifier IMEI 3rdParty 76% 85% 80% 26/55/19
Identifier IMSI 1stParty 33% 67% 44% 3/78/19
Identifier IMSI 3rdParty 69% 82% 75% 11/70/19
Identifier MAC 1stParty 83% 91% 87% 11/70/19
Identifier MAC 3rdParty 58% 78% 67% 23/58/19
Identifier Mobile Carrier 1stParty 78% 70% 74% 20/61/19
Identifier Mobile Carrier 3rdParty 92% 75% 83% 64/18/18
Identifier SIM Serial 1stParty 50% 50% 50% 2/81/17
Identifier SIM Serial 3rdParty 50% 88% 64% 8/75/17
Identifier SSID BSSID 1stParty 83% 56% 67% 9/74/17
Identifier SSID BSSID 3rdParty 53% 62% 57% 16/67/17
Location Cell Tower 1stParty 100% 100% 100% 2/76/22
Location Cell Tower 3rdParty 79% 73% 76% 15/63/22
Location GPS 1stParty N/A N/A N/A 0/77/23
Location GPS 3rdParty 70% 70% 70% 10/67/23
Location WiFi 1stParty 50% 100% 67% 1/77/22
Location WiFi 3rdParty 75% 75% 75% 12/66/22 Figure 2: Third party practices are discussed less frequently
Single Sign On: Facebook 56% 45% 50% 11/72/17
than first party practices. Given that users often have less
Table 4: Performance metrics of our system’s ability to de- opportunity to observe third party practices directly, it is un-
tect potential compliance issues on our test set of app/policy fortunate that they are not more widely discussed. A policy
pairs (n = 100). In the Support column, + is the number both affirming and denying a practice does not necessarily
of ground-truth positive instances of potential compliance imply a contradiction (e.g., “We disclose your phone num-
issues, - is the number of ground-truth negative instances, ber to advertisers, but not to data brokers.”).
and ? is the number of instances where missing ground truth
data from our app analyses makes it unclear whether or not
potential compliance issues exist. Which Practices are Described in Policies?
35.3% of the apps we analyzed had privacy policies.6 For
apps with privacy policies, Figure 2 depicts the occurrence
of policy statements relating to the practices we examine.
4 Privacy Compliance in the Play Store It can be observed that most practices are described only
infrequently; that is, a policy does not mention it at least
once. Further, the statements that are present typically affirm
that a practice is occurring. This finding reveals that users
Our large-scale analysis of free apps in the Google Play seem to be given little assurance of potentially objection-
Store provides us with a rich dataset for evaluating the state able practices not being performed (e.g., disclosing users’
of privacy in a substantial part of the Android ecosystem. phone numbers to third parties). Silence about privacy prac-
Here, we summarize our findings, with a focus on our pri- tices in privacy policies is problematic because there are no
vacy policy analysis. For a complete description of our find- clear statutory default rules of what the privacy relationship
ings, please see (Zimmeck et al. 2019). between a user and a service should be, in the absence of
explicit statements in the policy (Marotta-Wurgler 2015).
Prevalence of Potential Compliance Issues
Analyses at Scale
Our system detects potential compliance issues by compar-
ing the privacy policy analysis to the static analysis. A po-
tential compliance issue is detected when an app performs a
Designing and implementing a robust system to identify po- practice that is not described in the app’s privacy policy (if
tential compliance issues for large app populations presents the app even has a privacy policy). Note that when our sys-
challenges of scale. We address those with a pipeline of tem finds multiple privacy policies for a given app, it pools
distributed tasks implemented in a containerized software
stack. We performed our Play Store analysis from April 5
After the completion of the Play Store analysis we noticed a
6 to May 15, 2018. Out of 1,049,790 retrieved free apps, bug in our static analysis code. As a result, we re-performed the
1,035,853 (98.67%) were analyzed successfully. Of the static analyses and re-calculated all statistics. 135 additional analy-
apps which were not analyzed successfully, 1.03% failed ses failed, yielding a final total of 1,035,853 successfully analyzed
to download, 0.21% failed in the static analysis, 0.08% apps.
failed in the policy analysis, and 0.01% failed during our 6
This only counts English-language privacy policies: our sys-
re-analysis.5 tem does not identify policies in other languages.
belief that it is not their responsibility but the responsibil-
ity of the library developers to disclose to users the prac-
tices the libraries are performing. Some libraries’ terms of
services—for example, the Google Analytics Terms of Ser-
vice (Google 2018a)—obligate the developer integrating it
to explicitly disclose the integration in the developer’s pri-
vacy policy. However, this type of information transfer from
the third party via the developer to the user may be suscep-
tible to omissions and mistakes.
5 Conclusions
Natural language privacy policies are intended to commu-
nicate how a service collects, shares, uses, and stores user
data. However, as they are generally lengthy and difficult to
read, the average user often struggles to understand which
privacy practices apply. Leveraging natural language pro-
cessing techniques in the policy domain holds the promise
to extract policy content and convert it to a format that is
easier to comprehend. In this study, we reported on our de-
velopment of a three-tiered classification model to classify
a variety of privacy practices and their omissions in policy
Figure 3: The percents of apps which perform different prac- text. Compared to a monolithic classifier for a privacy prac-
tices, have policies that affirmatively describe practices as tice, using data type, party, and modality classifiers allows
performed, and have potential compliance issues. In this for economic use of training and test data—which is often-
graph, general descriptions of practices are counted with times expensive to obtain—as well as good performance.
specific descriptions. The classification model we are proposing here is an inte-
gral part of the Mobile App Privacy System (MAPS) (Zim-
meck et al. 2019). Many mobile apps are reliant on the col-
the practice descriptions discovered across all those poli- lection and use of a wide range of data for purposes of their
cies. This pooling has the effect of making our results rather functionality and monetization. MAPS presents one use case
conservative. One policy may disclose a particular practice for implementing the suggested privacy policy classification
while another policy discloses another practice, and together model. MAPS pairs our policy analysis with static anal-
they may cover all practices performed by the associated ysis of mobile apps to identify possible discrepancies be-
app. Overall, the average number of potential compliance tween the two and flag potential compliance issues. Our re-
issues per app is 2.89 and the median is 3. sults from analyzing 1,035,853 free apps on the Google Play
Figure 3 shows the percent of apps that perform vari- Store suggest that potential compliance issues are rather
ous practices and the respective percent of apps with po- common, particularly, when it comes to the disclosure of
tential compliance issues. The figure demonstrates that in third party practices. These and similar results may be of
many cases the performance of a practice is strongly asso- interest to app developers, app stores, privacy activists, and
ciated with the occurrence of a potential compliance issue: regulators.
if a practice is performed, there is a good chance a potential Recently enacted laws, such as the General Data Pro-
compliance issue exists as well. This result suggests a broad tection Directive, impose new obligations and provide for
level of potential non-compliance. Identifier-related poten- substantial penalties for failing to properly disclose privacy
tial compliance issues are the most common. Three different practices. We believe that the natural language analysis of
types of identifiers make up most potential compliance is- privacy policies, in tandem with mobile app analysis, for ex-
sues: cookies, device IDs, and mobile carriers. In particular, ample, has the potential to improve privacy transparency and
the use of device IDs may constitute a misuse for purposes enhance privacy levels overall.
of ad tracking (Google 2018b). In addition, there are also el-
evated levels of location-related potential compliance issues. Acknowledgments
15.3% of apps perform at least one location-related practice, We would like to thank Jongkyu Talan Baek, Sushain
and 12.1% of apps have at least one location-related poten- Cherivirala, Roger Iyengar, Pingshan Li, Shaoyan Sam
tial compliance issue. Li, Kanthashree Sathyendra, Florian Schaub, Xi Sun, and
For all data types, third party practices are more common Shomir Wilson for their help with this research. This study
than first party practices and so are third party-related po- was supported in part by the NSF Frontier grant on Usable
tential compliance issues. One reason for the prevalence of Privacy Policies (CNS-1330596, CNS-1330141, and CNS-
potential compliance issues for third party practices could be 1330214) and a DARPA Brandeis grant on Personalized Pri-
that app developers are unaware of the functionality of the vacy Assistants (FA8750-15-2-0277). The US Government
libraries they integrate. Perhaps they also hold the mistaken is authorized to reproduce and distribute reprints for Gov-
ernmental purposes not withstanding any copyright nota- McDonald, A. M., and Cranor, L. F. 2008. The cost of
tion. The views and conclusions contained herein are those reading privacy policies. I/S: A Journal of Law and Policy
of the authors and should not be interpreted as necessarily for the Information Society 4(3):540–565.
representing the official policies or endorsements, either ex- Mutchler, P.; Doupé, A.; Mitchell, J.; Kruegel, C.; and Vi-
pressed or implied, of the NSF, DARPA, or the US Govern- gna, G. 2015. A large-scale study of mobile web app secu-
ment. This work used the Extreme Science and Engineering rity. In MoST ’15.
Discovery Environment (XSEDE), which is supported by
National Science Foundation grant number ACI-1548562. Neisse, R.; Steri, G.; Geneiatakis, D.; and Fovino, I. N.
The authors acknowledge the Texas Advanced Computing 2016. A privacy enforcing framework for android applica-
Center (TACC) at The University of Texas at Austin for pro- tions. Computers & Security 62:257 – 277.
viding high performance computing resources that have con- Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.;
tributed to the research results reported within this paper. Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss,
R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.;
References Brucher, M.; Perrot, M.; and Duchesnay, E. 2011. Scikit-
Avdiienko, V.; Kuznetsov, K.; Gorla, A.; Zeller, A.; Arzt, learn: Machine learning in Python. Journal of Machine
S.; Rasthofer, S.; and Bodden, E. 2015. Mining apps for Learning Research.
abnormal usage of sensitive data. In 37th IEEE/ACM Inter- Ramanath, R.; Liu, F.; Sadeh, N.; and Smith, N. A. 2014.
national Conference on Software Engineering, ICSE 2015, Unsupervised alignment of privacy policies using hidden
Florence, Italy, May 16-24, 2015, Volume 1, 426–436. markov models. In ACL ’14.
Cranor, L. F.; Langheinrich, M.; Marchiori, M.; Presler- Ren, J.; Lindorfer, M.; Dubois, D.; Rao, A.; Choffnes, D.;
Marshall, M.; and Reagle, J. M. 2002. The Platform for and Vallina-Rodriguez, N. 2018. Bug fixes, improvements,
Privacy Preferences 1.0 (P3P1.0) specification. World Wide ... and privacy leaks – a longitudinal study of PII leaks across
Web Consortium, Recommendation REC-P3P-20020416. android app versions. In NDSS ’18.
Cranor, L. F.; Leon, P. G.; and Ur, B. 2016. A large-scale Reyes, I.; Wijesekera, P.; Reardon, J.; On, A. E. B.;
evaluation of U.S. financial institutions standardized privacy Razaghpanah, A.; Vallina-Rodriguez, N.; and Egelman, S.
notices. ACM Trans. Web 10(3):17:1–17:33. 2018. “Won’t somebody think of the children?” Examining
FTC. 2014. Complaint Goldenshores Technolo- COPPA compliance at scale. In PETS ’18.
gies. https://www.ftc.gov/system/files/ Sathyendra, K. M.; Wilson, S.; Schaub, F.; Zimmeck, S.; and
documents/cases/140409goldenshorescmpt. Sadeh, N. 2017. Identifying the provision of choices in
pdf. accessed: March 18, 2019. privacy policy text. In EMNLP ’17.
Google. 2018a. Google analytics terms of ser- scikit-learn developers. 2016a.
vice. https://www.google.com/analytics/ sklearn.feature extraction.text.tfidfvectorizer. http:
terms/us.html. accessed: March 18, 2019. //scikit-learn.org/0.18/modules/
Google. 2018b. Play Console Help. https:// generated/sklearn.feature_extraction.
support.google.com/googleplay/android- text.TfidfVectorizer.html. Accessed: March 18,
developer/answer/6048248?hl=en. accessed: 2019.
March 18, 2019.
scikit-learn developers. 2016b. sklearn.svm.svc.
Harkous, H.; Fawaz, K.; Lebret, R.; Schaub, F.; Shin, K. G.; http://scikit-learn.org/0.18/modules/
and Aberer, K. 2018. Polisis: Automated analysis and pre- generated/sklearn.svm.SVC.html. Ac-
sentation of privacy policies using deep learning. In USENIX cessed: March 18, 2019.
Security ’18.
Slavin, R.; Wang, X.; Hosseini, M.; Hester, W.; Krishnan,
Kumar, V. B.; Ravichander, A.; Story, P.; and Sadeh, N. R.; Bhatia, J.; Breaux, T.; and Niu, J. 2016. Toward a frame-
2019. Quantifying the effect of in-domain distributed work for detecting privacy policy violation in android appli-
word representations: A study of privacy policies. AAAI cation code. In ICSE ’16.
Spring Symposium on Privacy-Enhancing Artificial Intelli-
gence and Language Technologies. Solove, D. J., and Hartzog, W. 2014. The FTC and the new
common law of privacy. Columbia Law Review 114:583–
Libert, T. 2018. An automated approach to auditing disclo-
676.
sure of third-party data collection in website privacy poli-
cies. In WWW ’18. Story, P.; Zimmeck, S.; and Sadeh, N. 2018. Which apps
Liu, F.; Wilson, S.; Story, P.; Zimmeck, S.; and Sadeh, N. have privacy policies? In APF ’18.
2018. Towards Automatic Classification of Privacy Policy Tesfay, W. B.; Hofmann, P.; Nakamura, T.; Kiyomoto, S.;
Text. Technical Report CMU-ISR-17-118R and CMU-LTI- and Serna, J. 2018. I read but don’t agree: Privacy policy
17-010, School of Computer Science Carnegie Mellon Uni- benchmarking using machine learning and the EU GDPR.
versity, Pittsburgh, PA. In WWW ’18.
Marotta-Wurgler, F. 2015. Does “notice and choice” disclo- Wilson, S.; Schaub, F.; Dara, A. A.; Liu, F.; Cherivirala,
sure regulation work? An empirical study of privacy poli- S.; Leon, P. G.; Andersen, M. S.; Zimmeck, S.; Sathyendra,
cies. accessed: March 18, 2019. K. M.; Russell, N. C.; Norton, T. B.; Hovy, E.; Reidenberg,
J.; and Sadeh, N. 2016. The creation and analysis of a web-
site privacy policy corpus. In ACL ’16.
Yu, L.; Luo, X.; Liu, X.; and Zhang, T. 2016. Can we trust
the privacy policies of android apps? In DSN ’16.
Zhuang, Y.; Rafetseder, A.; Hu, Y.; Tian, Y.; and Cappos, J.
2018. Sensibility testbed: Automated IRB policy enforce-
ment in mobile research apps. In HotMobile ’18.
Zimmeck, S., and Bellovin, S. M. 2014. Privee: An archi-
tecture for automatically analyzing web privacy policies. In
USENIX Security ’14.
Zimmeck, S.; Wang, Z.; Zou, L.; Iyengar, R.; Liu, B.;
Schaub, F.; Wilson, S.; Sadeh, N.; Bellovin, S. M.; and Rei-
denberg, J. 2017. Automated analysis of privacy require-
ments for mobile apps. In NDSS ’17.
Zimmeck, S.; Story, P.; Smullen, D.; Ravichander, A.; Wang,
Z.; Reidenberg, J.; Russell, N. C.; and Sadeh, N. 2019.
MAPS: Scaling privacy compliance analysis to a million
apps. In PETS ’19.