The processing of personal data -and therefore data protection legislationis an essential facet of a modern economy. Personal data is widely associated with a modern business model made of Internet-based services offered free of charge and whose revenues come from the collection, the processing and the use of personal data for advertising purposes, but it comes into play also with more traditional business models, such as the product market, which are evolving to e-commerce, or tailoring their offers to individual customers.
The legal landscape of the protection of personal data within and outside the European Union (EU) has been redesigned by the amplitude of the material and territorial scope of the General Data Protection Regulation (GDPR) [5,25,21], as had already happened, to some extent, with the previous Directive [9].
This new landscape, coupled with the heavy fines that supervisory authorities are entitled to issue in case of violation, calls for a need to ensure that data processing activities (and tools thereof) comply with the GDPR. Data controllers and processors could therefore take advantage from tools helping design and verify compliance, thus diminishing their risks of violating provisions and incurring into fines. Such tools could also help companies build, in an automated or computer-assisted way, their case in response to one-time casuistic decisions emanated by supervisory authorities and courts against them.
A critical facet for such automation is to be able to build executable rules for a computer-assisted assessing compliance. In previous research, the authors have proposed a complete model of the GDPR for legal reasoning and legal compliance [1,17,16,15]. This model comprises three components: the legal text in Akoma Ntoso format, an ontology of legal concepts concerning privacy and data protection, and a knowledge base of data protection rules in LegalRuleML format. This last component, called the Data Protection Regulation Compliance (DAPRECO) Knowledge Bas Knowledge Base, is the most critical: it contains the bulk of provisions written as logic rules that can be used e.g., to check whether certain practices (themselves also formalized) are aligned with the provisions. Consequently, the Knowledge Base needs to be adequately (i.e., legally) validated before it can perform in a real-world environment. This pragmatical strand is demanded, since "for developers, as contrasted to researchers, the issue is not whether the resulting rule base is complete or even accurate or selfmodifying -but whether the rule base is sufficiently accurate to be useful" [3] when it is moved out from the research laboratory and into the marketplace [24].
However, as is widely acknowledged in literature [19,23,11], testing Legal Knowledge Based System (LKBS) is a difficult task. It is more difficult even than software testing in general because approaches reveal coder-dependency and it is complex to emulate the "art-of-the-experts" [4]. With ongoing maturity in the field of AI and Law and Legal Knowledge-Based Systems, the need for an easily accessible and interdisciplinary validation methodology comes into play [10].
The concept of validation refers to the determination of the correctness of the system with respect to user needs and requirements [23]. Legal validation is thus "needed to verify the correctness of the output of the system in relation to the knowledge of the legal domain it covers", "the guarantee of the one-to-one relation between analysis and representation" [12]. Such a method would assist legal professionals framing an evaluation of legal knowledge-based systems and help IT experts understanding the validation requirements of legal professionals [11].
As "algorithmic representations of law are typically very poor as regards their transparency", "one cannot begin to devise an algorithm to apply legal provisions without determining first its intended purpose and by whom it will be used" [22]. Therefore, validating a legal model requires that the formalization used is understandable, accessible. Consequently, the methodology should be driven by usability considerations in the adopted criteria, and validation tests (through user acceptance surveys or questionnaires) [23].
In the particular domain of modelling the GDPR, a considerable effort has been reserved to represent the Regulation's provision in a logic formalism, precisely the Reified Input/Output (RIO) logic [20]. As we will recall in Section 3, the logic formulae refer to concepts that belong to a legal ontology for data protection, that is, the Privacy Ontology (PrOnto) ontology which is the result from an interdisciplinary effort meant to provide legal soundness. But of course referring to an ontology is not sufficient to ensure legal soundness in a reasoning process for many reasons, among which that the ontology has been devised for concepts relevant in data privacy but not for the specific context of the GDPR and thus new concepts may need to be expressed; besides, certain legal interpretation are anticipated e.g., in the choice of the formula's functions or when deciding that certain articles express an obligation or a permission. We will discuss further the critical points of a formalization exercise but, in short, it should be evident that formalizing articles in a logic formalism requires a legal supervision. Postponing any legal validation until the whole GDPR is translated into RIO formulae, as it would happen if we were awaiting to have any output of the enabled logical reasoning, is a procedure prone to errors. Finding and removing the cause of some unsound conclusions would be also a very expensive step if left only at the end, quite likely inspiring distrust in the whole framework. A more agile process is advisable to verify for legal soundness, assisting who is responsible for the formalisation of the GDPR incrementally and concomitantly during the modelling work.
Pursuant to this, we discussed in [2] such an agile methodology, proposing also a solution for a further problem that arose in this case: how to let expert in law understand the logic formula, which are usually embedded in a machine readable but quite human incomprehensible LegalRuleML, in such a way to collect informed feedback from them about the legal soundness of the formalization. The solution is an intermediate representations, devised human-readable, of a RIO logic formalization of two GDPR articles. A usability experiment involving legal domain experts, consisting in assessing the readability and understandability of the human-readable representation compared with the original LegalRuleML version and with another control format, brought evidence that the former suits the purpose of being used for an interdisciplinary legal validation.
The customizable human-readable representation has been assessed as understandable, increasing our confidence on it being an eligible candidate to validate the formalized GDPR articles. In this work we proceed further and show that the methodology is effective in gathering feedback of legal experts on the legal validity of the representation of the GDPR articles, so as to provide quality assurance of our methodology as a whole.
Some discussion within the AI and Law community [23,11,10] -specifically amidst the Proceedings of the International Conference on Artificial Intelligence and Law works (ICAIL), and later through the Journal of Artificial Intelligence and Law contributions (JAIL), -concerned qualitative evaluation methodologies suitable for legal domain systems/techniques, and the best practices through which AI and Law researchers could frame the assessment of the performance of their works, both empirical and theoretical. For example, performance evaluation is emphasized and compared to known baselines and parameters, using publicly available datasets whenever possible [7,8].
A set of six categories was compiled to define the broad types of evaluation found therefrom. They include the following assessments: i. Gold Data: evaluation performed with respect to domain expert judgments (e.g., classification measurements or measures on accuracy, precision, recall, F-score, etc.); ii. Statistical : evaluation performed with respect to comparison functions (e.g., unsupervised learning: cluster internal-similarity, cosine similarity, etc.); iii. Manual Assessment: performance is measured by humans via inspection, assessment, review of output; iv. Algorithmic: assessment made in terms of performance of a system, such as a multi-agent system; v. Operational-Usability: assessment of a system's operational characteristics or usability aspects; vi. Other : those systems with distinct forms of evaluation not covered in the categories above (task-based, conversion-based, etc.). In our case, we combined the following types of evaluation: gold data (i.), manual assessment (iii.) and operational-usability (v.).
Some authors [10] developed the Context Criteria Contingency-guidelines Framework (CCCF) for evaluating LKBS. Besides considering evaluation context and goals, they also propose evaluative criteria and guidelines alike. In this framework, the quadrant criteria pertinent to the purposes of this paper are herewith mentioned. The User Credibility quadrant refers to credibility and acceptability of a system at the individual level. It comprises three main branches associated with user satisfaction, utility (usefulness or fitness for purpose) and usability (ease of use). The usability branch is further decomposed into branches associated with operability, understandability, learnability, accessibility, flexibility in use, and with other human factors and human computer interface issues. The Verification and Validation criteria quadrant refer to knowledge base validity, including knowledge representation and associated theories of jurisprudence, inferencing, and the provision of explanations.
The validation phase of legal modeling by domain legal experts -driven by operational usability assessments -is also mentioned in the methodologies referring to ontological expert knowledge evaluation. For example, the Methodology for Modeling Legal Ontologies (MeLOn) [14] offers evaluation parameters consisting in completeness, correctness, coherence of the conceptualization phase and artifact reusability. Usability concerns were considered in an experimental validation of a legal ontology by legal experts, the Ontology of Professional Judicial Knowledge (OPJK), described in [6]. This model was validated in a two-step process. First, the evaluators answered a questionnaire whereby they expressed their opinion on their level of agreement towards the ontology conceptualization and provided suggestions for the improvement thereof. Then an experimental validation based on a usability questionnaire followed, the System Usability Scale (SUS), tailored to evaluate the understandably and acceptance of the contents of the ontology. This evaluation questionnaire could offer rapid feedback and support towards the establishment of relevant agreement, shareability or quality of content measurements in expert-based ontology evaluation. An evaluation methodology based on Competency Questions (CQs) [18] was built to evaluate the transformation of legal knowledge from a semi-formal form (Semantics Of Business Vocabulary And Rules -Standard English (SBVR-SE)) [13] to a more structured formal representation (OWL 2), and to enable cooperation between legal expert and knowledge IT expert in charge of the modelling in logic formalism. Ontology quality criteria were accounted for.
Although the legal formal framework target of this work's analysis (i.e., the DAPRECO Knowledge Base) refers and is strictly bound to a legally validated ontology (i.e., the PrOnto) an argument for its legal validity cannot follow only from the validity of the ontology of reference. It requires a more comprehensive analysis and we believe that both qualitative evaluation methodologies and certain criteria from the CCCF are required. Ontologies are in fact about concepts, data, and entities and any validation strategy of them is inevitably about assessing the legal qualities of those objects. Formal models for legal compliance, such as the DAPRECO Knowledge Base, model also the logical and deontic structure of a legal text, its temporal aspects and, as when the formalism allows multiple conflicting interpretations, as the DAPRECO Knowledge Base does, it includes structural elements to allow defeasible reasoning. The validation assessment should take these elements into account.
Thus, the necessity of an integrated approach, which additionally should also acknowledge an operational-usability assessment, since the legal validity of the DAPRECO Knowledge Base logic formulae have to be validated by people who are not experts in logic.
This work leverages on our previous work [2] which discusses how to take the formalization of GDPR provisions expressed in RIO logic [20] and how to extract from them pieces of information that a legal expert will process of legal validity. This, in fact, means to answer a questionnaire whose questions are meant to provide feedback on specific quality linked to the legal validity, such as completeness, coherence, and conciseness (see later). We will refer to this synthetic digest of an otherwise specific logic formalism as human-readable representation of a RIO formula.
Empirical validation using tailored constructed questionnaire is a very useful quantitative indicator of user acceptance [26]. In this case, where users are lawyers, the questionnaire has been designed with the purpose of having legal feedback on the quality of the legal interpretation in the RIO formulae. The questionnaire is build around six questions reported in Table 1.
Table 1. The questions used.
Does the deontic modality (obligation/permission/other) of the formula coincide with the modality of the GDPR articles? q2 Does the formula capture all the important legal concepts? q3 Does the formula capture all the important legal relations? q4 Is the interpretation given by the model correct? q5 Is the interpretation complete? q6 Is the interpretation to the point?
They are tailored to check for the following legal validation qualities: Accuracy (q 1 ); Completeness (q 2 − q 4 ); Consistency (q 5 ); and Conciseness (q 6 ).
We now give two examples of human-readable representations accompanying the questionnaire. Both examples will be referred later in this paper.
Example 1. We refer to Article 7.1 of the GDPR, which reads: "Where processing is based on consent, the controller shall be able to demonstrate that the data subject has consented to processing of his or her personal data" Without entering into any detail on its modeling (the reader can refer to [20] for it), RIO formula expressing this provision is the following: Names like nominates and PersonalData are either chosen by the IT expert in charge of the modelling in logic formalism or are taken from the PrOnto [17,16,15]. PrOnto is one of the three essential components along with the Akoma Ntoso representation of the legal text and the formulae in RIO logic. When connected together the three components form the complete model of the GDPR. The formula expresses an obligation, and that is what the O at the rightmost end Equation ( 1) is supposed to mean. Equation ( 1) is not however what a user would see. Stored to be machine-readable, the formula is written in its LegalRuleML version. An excerpt of this version is given below.
There is a processing, which has a purpose authorized by a consent given by a data subject, and that is what a processor, whom a controller controlling the personal data nominates, does on personal data of the data of the data subject.
Whenever there is a processing, which has a purpose authorized by a consent given by a data subject, and that is what a processor, whom a controller controlling the personal data nominates, does on personal data of the data of the data subject then the controller is obliged to able to demonstrate that "data subject gave consent".
Example 2. We refer to Article 5.1(a) of the GDPR, which is worded as follows:
"Personal data shall be a) processed lawfully, fairly and in a transparent manner in relation to the data subject ('lawfulness, fairness and transparency');"
The output of the re-interpretation of the RIO formula (here omitted for reasons of space), transform the LegalRuleML as follows: The output of the hand-made processing is shown in Table 3.
There is a processing done by a processor on personal data of the data subject, and a controller, which is who controls the personal data; he nominates the processor. The controller describes (how) he implements a measure.
Whenever there is a processing done by a processor on personal data of the data subject, and a controller, which is who controls the personal data and who nominates the processor. Then before that moment must be that the controller describes (how) he implements a measure that is what causes lawfulness, fairness and transparency of the data processing related to the data subject.
Can a legal validator reading the human-readable representation give useful feedback to the modeller? Answering this question is the goal of this paper.
Rephrased, the goal is to show that the "Validation" phase of the methodology presented in [1] (see Figure 1) is effective, i.e., helpful feedback can be collected for the IT expert formalizing the GDPR with RIO logic. The starting point is the human-readable representation of Articles 5.1(a) and 7.1 of the GDPR. The "Check" action (see Figure 1) has been implemented by gathering a set of four validators, all jurists knowledgeable of data protection law, and by asking them to answer questions q 1 − q 6 in Table 1.
Evaluators were told to compare the meaning of the formulae, as expressed in the human-readable representation of the RIO logic, with the legal interpretation that them legal experts would give to the original articles in the GDPR.
We also asked them a few questions meant to reveal how much understandable for them is the human-readable format, before they start using it. General understandability of the format has been discussed elsewhere [1], but here the assessment is meant as a trust measure over the expert's answers. However, all our evaluators confirmed they have found the human-readable format understandable. From those trusted answers, we therefore compile a few recommendations. This is the "Generate Feedback" step in Figure 1.
Questions q 1 − q 6 (see Table 1) are yes/no questions, but in the questionnaire we additionally invited our checkers to motivate their answers and to pinpoint further whatever observation they valued meaningful. We collected eight documents with such written answers and comments which we reviewed and summarized. The following table resumes the findings, wherein we report the comments whenever the answer to the question was 'no', indicating that someone found some pertinent issue. Table 4 shows that legal experts were able to give feedback on all the factors about the quality of the legal interpretation in the logic formalization of the articles. Even if the input to provide to the IT expert is not yet straightforward, a few highlights clearly emerge.
For instance, all experts easily understood and confirmed the deontic modality and agreed on the formulas be capturing all the legal concepts and relations (see Table 5). But is from the analysis conferred to the provided comments that we are able offer a broader spectrum, for they refer to the above surveyed criteria and also to other (non-surveyed) related criteria. Comments -in Completeness like "it was complex to capture the legal concepts within the structure of the formula"; comments in Consistency like "It refers to the implementation and description of a measure that it is hard to understand; "It is redundant and restates concepts already present at previous articles", and comments in Conciseness like "'Obliged to be able to' sounds weirds" -clearly indicate uneasiness about the way in which the formula have been structured; such comments may lead to a better formalization, for instance, stating certain contextual facts as a common premise valid for all the GDPR's articles without repeating them each time in each article.
One evaluator, in particular, has mentioned "Interchanged roles for the controller and the processor" in Consistency. Even if that is stated in the Context of the human-readable table, the evaluator was probably induced in error/confused by the excess of information provided. Further analysis is of course required. Ex-
It was complex to capture the legal concepts within the structure of the formula; It is missing the obligation: "the processing must be fair, lawful, transparent" It was complex to capture the legal concepts within the structure of the formula;
Interchanged roles for the controller and the processor; The interpretation is complex. It refers to the implementation and description of a measure that it is hard to understand; I can read/understand the model, but I think it does not faithful to the article's meaning;
The reference to consent should be enhanced, namely regarding the requirements concerning the burden of the proof; "Shall" is not captured;
The formula mentions "implement" "describe" not expressed in the article; "implement measure" is not expressed in the article; "Obliged to be able" sounds weird;
It is redundant and restates concepts already present at previous articles; tracting from the non-structured comments valid input for the IT expert has to be left as future work, as we comment in the following section.
This paper leverages a methodology that advocates an interdisciplinary validation of a representation of the GDPR articles in a logic formalism (i.e., RIO logic) to pursue quality, accountability, and transparency within. One important output of the methodology is the production of feedback derived from the involvement of legal experts, while assessing the quality of the legal interpretation that IT experts may instill in the formalization of the GDPR. This work has gathered evidence that such step is feasible. As a proof-of-concept, a small number of legal experts has been asked to answer six questions with the purpose of collecting comments about how two logic formulae, modelling Articles 5.1a and 7.1 of the GDPR, are complete, accurate, concise, and consistent in reflecting the legal meaning of the articles. Several comments have been collected. Although a thorough analysis thereof requires more time -an involvement of a larger group of expert checkers is also advisable-we were able to identify a few issues of relevance using which the IT expert can review the formalization work.
Several challenges await us in the near future. We need to improve scalability in producing a human-readable representation of the RIO formulae: it is currently done manually, starting from the pre-processed version. This is already more readable than the original LegalRuleML version and give us confidence that the work to produce a natural language analysis break-up table can be automatized. This step done, a forth bringing process will consist in streamlining the validation of the RIO formalization of the GDPR as a whole. This likely requires to set up Table 5. Inter-validators agreement on answering 'yes' to the questions an application where the work of the IT expert can be suitably translated in to the human-readable format and offered for on-line checking to a group of legal testers, which may also vary, providing feedback that the IT expert can take into consideration until a good quality of legal interpretation is assessed for the formulae.
Concomitantly, there is a need to define together with the legal experts a more complete set of qualities and possibly a few metrics, which we can quantify and define criteria on the legal quality of the formalization. In Section 2 we pointed out possible metrics, and in this paper we have assessed a few (completeness, consistency, conciseness in Section 4), but a wide and systematic investigation of the state-of-the-art in this topic has not been done yet. The quadrant criteria presented in [10] also merits attention. This may lead to a revision of the current human-readable model.
Lenzini are supported by the FNR CORE project C16/IS/11333956 "DAPRECO: DAta Protection REgulation COmpliance".
< r u l e m l : i f> <r u l e m l : E x i s t s> <r u l e m l : A n d> <r u l e m l : A t o m> <r u l e m l : R e l i r i =" r i o O n t o : R e x i s t A t T i m e " /> <r u l e m l : V a r k e y r e f=" : a 1 " /> <r u l e m l : V a r key=" : t 1 ">t 1</ r u l e m l : V a r> </ r u l e m l : A t o m> . . . <r u l e m l : A t o m> <r u l e m l : R e l i r i =" p r O n t o : D a t a S u b j e c t " /> <r u l e m l : V a r k e y r e f=" :w " /> </ r u l e m l : A t o m> . . . </ r u l e m l : A n d> </ r u l e m l : E x i s t s> </ r u l e m l : i f> <r u l e m l : t h e n> <r u l e m l : E x i s t s> <r u l e m l : A n d> <r u l e m l : A t o m> <r u l e m l : R e l i r i =" r i o O n t o : R e x i s t A t T i m e " /> <r u l e m l : V a r k e y r e f=" : e a " /> <r u l e m l : V a r k e y r e f=" : t 1 " /> </ r u l e m l : A t o m> . . . </ r u l e m l : A n d> </ r u l e m l : E x i s t s> </ r u l e m l : t h e n> </ r u l e m l : R u l e>
In [1], we described a parser that reads LegalRuleML and returns an itemized structured representation of the formula whose intended meaning has not been changed and is preserved in the translation. This version renders a cleaner version without all those XML based tags which are quite hard to process by a human reader. Applied to the LegalRuleML of (1), it results in the following text: The words capitalized and in bold are concepts from the PrOnto ontology. The words in bold non-capitalized are relations introduced by the IT expert. Although this format still requires some mental effort to be read, it is at least human-processable;
The human-readable representation is built from the output of the parser through a manual post-processing. That of Article 7.1 is shown in Table 2.