The ubiquitous availability of the Internet results in a huge number of apps, software and online services with accompanying contractual agreements in the form of `terms of use' and `privacy policy'. Although everyone is exposed to such consent forms, the majority tend to ignore them due to their length and complexity. In this thesis, we focus on interpretation of contractual agreements for the bene t of endusers. By applying text mining and semantic technologies, we develop an approach that extracts important information and retrieves helpful links and resources for the better comprehension. Our approach is based on ontology-based information extraction and machine learning and delivers the unpleasant consent form in a user friendly and visualized format. The evaluation shows that although semi-automatic approaches lead to information loss, they save time and e ort by producing instant results and facilitate the end-users' understanding of legal texts.
The increasing availability of online services and mobile apps has led to a huge
proliferation of terms and conditions regulating their use. In the digital age
everyone is exposed to such terms, and in their majority this constitutes ordinary
people with limited to no knowledge of legal terms. The problem arises when
people ignore the consent forms due to their length and complex terminology.
In a recent study, \The biggest lie on the Internet", 543 students were asked
to agree to a privacy policy and terms of use in order to join a ctitious social
network [
In order to facilitate the process of digesting terms and conditions for regular end-users, we consider applying text mining and use of domain ontologies 1 http://www.prnewswire.co.uk/news-releases/skandia-takes-the-terminal-out-ofterms-and-conditions-145280565.html to provide visualized summaries. The approach considered is broadly applicable to other forms of text-based contractual agreements. However, in this thesis we speci cally focus on terms of use (aka. End-User License Agreement or EULA) and privacy policies, since they have the broadest impact and a ect everyone. Our research questions which will be answered in the sequel, speci cally include: 1) Dose text mining techniques for extracting and summarizing important information from consent forms lead to information loss? and 2) Does our approach need less time and e ort for contractual agreements comprehension? 2
Consent forms such as terms of use and privacy policies have clear structure and
terminologies. Therefore, OBIE is a tting method for processing such texts,
since the mappings between natural language text and machine-understandable
conceptualizations is more straightforward. OBIE uses an ontology to guide the
IE pipeline and annotates the text with the ontology concepts. In recent years,
along with the increasing emergence of domain ontologies, OBIE has gained a
lot of interest. According to a survey of OBIE applications [
Our literature review covers speci cally license agreements and privacy policy
studies. Tl;drLegal5 is an online service that uses a manual and crowdsourced
way to present a summary of popular EULAs. Furthermore, NLL2RDF is a rst
attempt which employs NLP and ML techniques to generate RDF expressions
of license agreements [
Some e orts have speci cally studied privacy policies [
After a thorough literature review covering terms of use (or EULAs) and privacy policies, we considered di erent approaches to identify the ones that are more suitable for each type of agreement and the in/availability of training corpora. For privacy policy analysis, we rely on the OPP-115 dataset to apply supervised ML and train a reliable model. In contrast, there is no annotated corpus for terms of use, and the creation of one poses a major challenge because they don't follow a speci c structure and their scope is generally broader. Depending on the type of an asset (software, website, digital products, etc.), the terms of use di er signi cantly. They may contain copyright conditions, speci c rules on accessing the service, intellectual property rights and various other content. However they all share a common characteristic: they are written using legal terminology | from which it is able to extract a common structure. Based on this assumption, we apply OBIE for extracting pre-de ned classes of information.
Having investigated the existing EULA ontologies and vocabularies, we have
chosen ODRL as the main ontology for our OBIE pipeline. It is speci ed in
W3C recommendations and has also demonstrated the highest community
endorsement. Although the focus of ODLR is digital content, it is broad enough
to cover di erent types of resources. Bene ting from Permission, Prohibition
and Duty classes and their properties (e.g., hasAction), we de ne tailored rules
utilizing GATE JAPE grammar [
Although OBIE is a standard approach, there has been no prior study utilizing OBIE for EULAs. From this point of view our application is new. Furthermore, since we are bene ting from a standard `model' of the domain, there is a huge potential to better structure similar documents along the same taxonomy. Moreover, having a vocabulary to cover such legal texts can become a standard for structuring also new documents (and not just the existing ones). 4
Our framework consists of two separate modules: EULAide is responsible for processing license agreements (or terms of use) and is based on OBIE; and KnIGHT assigns pre-de ned categories to a privacy policy paragraphs and is built upon a supervised ML approach. Figure 1 shows the high-level architecture of our framework and each module is presented in the following subsections. EULAide extracts important excerpts from EULAs. As shown in the picture, a pre-processing module performs common NLP tasks: tokenizer, sentence splitter, POS tagger, root nder and a text- le gazetteer which contains important keywords from EULAs (e.g., di erent synonyms for the terms `license' and `asset'). The pre-processed EULA will be ingested to the OBIE pipeline, which contains JAPE hand-coded grammar rules based on ODRL community speci cation documentation10. Till now, we have implemented 15 rules, some of which are: i) Ontology-based annotation: separates all ontology-derived Action instances into DutyAction & PermProhAction based on the ontology speci cation; ii) ExtractPermWords : identi es the important keywords for permissions detection, e.g., may, can, grant, permit, etc.; iii) ExtractPermission: extracts the whole sentence, if the pattern is matched. Table 1 shows the steps towards extracting of a sample permission. After the pre-processing phase, rst the text- le gazetteer produces two annotations: License & Asset. Second, the ontology-based annotation generates PermAction annotation. Third, the extractPermWords rule res and PermWord annotation is created. Finally, extractPermission detects the whole sentence as a Permission. 10 https://www.w3.org/TR/odrl-vocab/
The summarization component clusters the similar extracted excerpts and creates a short description for each cluster. Figure 2 shows an example of EULAide output. The number of extracted excerpts by OBIE pipeline is 14, whereas the summarization module has reduced the number of clusters to 9.
In order to evaluate the e ciency of EULAide we conducted an experiment to identify if the solution enables end-users to invest less time and e ort to su ciently comprehend it. At the same time, we wanted to identify the tradeo between the added support and the information loss expected when applying semi-automatic IE and summarization. As a rst step, a corpus containing twenty EULAs in their natural language texts was compiled. Two annotators familiar with EULA texts annotated the corpus independently following an introduction to the relevant ODRL concepts. The Inter-Annotator Agreement (IAA) between two annotators is 90%, which indicates the production of a reliable gold standard. To identify the cost of IE-in icted information loss, a legal expert designed 5 multiple choice questions for four EULAs (e.g., 20 in total). All questions are related to Permission, Prohibition & Duty. In the last step 6 volunteers from the university campus (postgraduate students and sta ) were required to answer these questions using two methods: i) reading the EULA in full text and ii) utilizing EULAide. The results are brie y presented in section 5. Privacy policies are legal documents stipulating how companies will gather, manage and process customer data. They are legally required for any service that uses, maintains or discloses data that can be used to identify an individual. In contrast to EULAs, privacy policies must comply with a smaller set of legislation, i.e., data protection laws. This focus enables us to perform more speci c analysis and check compliance against speci c data protection regulation. For such contractual agreements, we employ a deep learning approach utilizing an existing corpus. The OPP-115 dataset is divided into paragraphs, each of which includes annotations from three legal experts. There are two types of annotations: at the top level each paragraph is labeled with one (or more) pre-de ned classes; and at a lower level a class may contain speci c attributes. For example, the top level category \User Choice/Control" can be narrowed down to: choice type, choice scope, personal information type, purpose. We are working on training KnIGHT with OPP-115 and extract top level classes and lower level attributes from a policy. Having this structured information from privacy policies, we are able to: 1) check their completeness according to the legislation; 2) measure their risk factor by analyzing the values of attributes; 3) map the top level categories to data protection laws for more advanced users (legal experts, data o cers, etc.).
The rst two goals target regular users as the intended audience, whereas the third one is more suitable for experts. Although the OPP-115 consists of policies de ned by American companies, the top level categories can be mapped to GDPR. For instance, the category \First Party Collection/Use" is related to Article 13, `Information to be provided where personal data are collected', or \User Access, Edit & Deletion\ category can be linked to Article 16 & 17 (`Right to Rectification/Erasure'). The mappings can be as general as a whole article or as detailed as a speci c paragraph. For the evaluation of KnIGHT, the rst two directions will be assessed using the OPP-115 as a gold standard, while the mapping accuracy should be assessed by experts. 5
In this section, we will only discuss results from experiments seeking to evaluate the support provided by EULAide to make sense of legal agreements (terms of use). The evaluation of our privacy-policy (training data-based) approach is still in planning stage. In order to measure the performance of the OBIE pipeline, we have used our compiled gold standard based on the manually-annotated examples (excluding the 10% disagreement in the IAA exercise). The evaluation results are shown in tables 2a & 2b. The ontology enhancement was a feedback cycle during which we improved domain-speci c coverage by adding additional instances (around 50), with the support of a legal expert. Considering the complexity of EULAs and the 90% agreement observed between human annotators, the results indicate that our OBIE method yields useful results and is feasible. The (90 - 72)% information loss comes from the incomplete set of grammar rules and ODRL instances coverage. Expanding the ontology with more concepts will allow us to de ne more rules and will eventually increase the system accuracy.
F1
Tables 3a & 3b present results from the previously described EULAide usability experiment. Phase1 and Phase2 indicate di erent phases when answering the multiple-choice questions. In the rst phase participants read EULAs (either in their full text or utilizing EULAide) and answered the questions using their memory. In the second phase they were allowed to use search tools for unanswered questions. The rationale behind this setting was to recreate the baseline method for users to check and read policies without any tool. Thus, we sought to identify how well regular people can remember policies and how fast they can search for information in an EULA. In practice, when one is agreeing with terms, this process should be followed so as to be aware of the rights and regulations. Our results verify our initial hypotheses, i.e., even though EULAide is e ected by a (12 - 1.5 = 10.5)% information loss, it considerably saves time and e ort spent by users to arrive to a similar level of understanding. Finally it should be stated that although due to funding restrictions the number of selected EULAs and participants was the bare minimum required for an experiment of this kind, the results were su cient to indicate the value in extending and improving our approach.
Reading Phase1 Phase2 EULA Full Text EULAide 1185 315 75 72 152 77 (a) Average time (In Sec.)
Correct Incorrect EULA Full Text EULAide 67 62 8 15
Unanswered in Phase1 Phase2 Phase2 Phase2 Correct Incorrect Unanswered 18.5 5 1.5 6.5 4.5 12 (b) Average percentage of questions results (%)
This thesis tackles the important issue of di cult-to-read legal documents and investigates automated methods for the bene t of end-users. The experiments conducted con rm the complexity of the task and the subjectivity of human judgment. Somewhat counter-intuitively, we observe that agreement between the experts is generally harder to achieve than between average users. This is probably due to the experts' higher understanding and ability for a more critical inspection of legal texts. A constant non-technical challenge in our e orts is to attain commitment from legal experts on a voluntary basis. Despite the challenges and di culties, our results so far indicate that NLP techniques combined with OBIE and ML can be very useful to support legal text comprehension and that with su cient funding broader experiments can be carried out.
I would like to thank Simon Scerri, Soren Auer and Jens Lehmann for their guidance and fruitful discussions during the development of this doctoral work.