=Paper=
{{Paper
|id=Vol-2385/paper8
|storemode=property
|title=Modelling Norm Types and their Inter-relationships in EU Directives
|pdfUrl=https://ceur-ws.org/Vol-2385/paper8.pdf
|volume=Vol-2385
|authors=Ilaria Angela Amantea,Luigi Di Caro,Llio Humphreys,Rohan Nanda,Emilio Sulis
|dblpUrl=https://dblp.org/rec/conf/icail/AmanteaCHNS19
}}
==Modelling Norm Types and their Inter-relationships in EU Directives==
https://ceur-ws.org/Vol-2385/paper8.pdf
Modelling Norm Types and their Inter-relationships in EU
Directives
Ilaria Angela Amantea, Luigi Di Caro, Llio Humphreys, Rohan Nanda and Emilio Sulis
University of Torino
Torino, Italy
{amantea,dicaro,humphrey,nanda,sulis}@di.unito.it
ABSTRACT This research is concerned with normative provisions and recitals
EU directives are deliberately imprecise to allow Member States to in EU directives. While recitals lack the prescriptive status of nor-
fulfill the objectives in their own way. It follows that deontic norms mative provisions proper (their main purpose being to provide the
in EU directives cannot be properly understood without considera- wider context), they have been shown to yield some influence on
tion of the motivation behind those norms, as well as consideration the interpretation of normative provisions. The research questions
of how other provisions, or even other laws, may affect the scope of this paper are therefore:
or effectiveness of the norm in question. This paper seeks to model (1) What kind of norms are present in normative provisions and
norm types and their inter-relationships. We envisage that this recitals?
preliminary analysis may help to develop an automated system to (2) What kind of links are there between norms, be they norma-
find norms that are related in different ways in order to help legal tive provisions or recitals?
professionals interpret laws for specific cases.
Section 2 describes the challenges of legal reasoning, European
KEYWORDS law in the global context, and the nature of EU directives. Section
3 introduces our classification of norm types and section 4 our
Natural language processing, Law, Machine learning, Recitals, Eu-
classification of link types. Section 5 provides issues for discussion,
ropean legislation, purposive interpretation, cosine similarity
and section 6 outlines preliminary ideas for automated classification.
Section 7 describes related work, and section 8 ends the paper with
1 INTRODUCTION conclusions and future work.
Norms are not “legal flowers without stem or root” [39], page 27. A
normative provision almost always has to be read in the context
2 BACKGROUND
of other norms. Generally, the law has a holistic character (cf. [43]
and [19] among others) which emerges from a network of legal 2.1 The Nature of Legal Reasoning
documents. This means that at times the meaning of legal norms The problem of vagueness pertains to all human reasoning, not
emerges not from single parts of a normative provision but from a just legal interpretation. As Sainsbury [41], page 324, explains:
wider legislative corpus. Indeed, it is arguable that the context is ‘Vagueness gives rise to borderline cases. Think, for example, of
even wider including parliamentary debates, legal common practice, a colour spectrum. There are clear cases of red and clear cases of
and doctrinal interpretation. As such, legal interpretation takes orange, and in between there are borderline cases: shades which
much effort, and can be helped by automated efforts to find such we don’t feel inclined to classify either as red or as orange.’
links. Travis [47], pages 171-172, shows how context can influence the
Less researched perhaps are important links between normative meaning of even a simple question such as whether a leaf is green:
provisions in the same piece of legislation. Our manual analysis of ‘Suppose a Japanese maple leaf, turned brown, was painted green
a lengthy European Union (EU) directive revealed not only very for a decoration. In sorting leaves by colour, one might truly call
different kinds of norms, but also different kinds of relationships this one green. In describing leaves to help identify their species, it
between norms. While EU directives have particular characteris- might, for all the paint, be false to call it that. ’
tics (outlined in the next section), the general point remains that In the legal context the most famous example of such phenomena
understanding of particular normative provisions can be greatly is the vehicle in the park scenario outlined by Hart [20], page 607:
enhanced by reading them in conjunction with those norms that ‘A legal rule forbids you to take a vehicle into a public park. Plainly
influence them in some way. Needless to say, not all normative this forbids an automobile, but what about bicycles, roller skates,
provisions are of such relevance, and our research goal is therefore toy automobiles? What about airplanes?...We may call the problems
to help legal researchers find those provisions by semi-automated which arise outside the hard core of penumbral instances “problems
means. It is hoped some of the insights in this paper about the of the penumbra”...If a penumbra of uncertainty must surround all
nature of norm types and relation between them will pave the way legal rules, then their application to specific cases in the penumbral
to automated detection of different kinds of norms and relations. area cannot be a matter of logical deduction, and so deductive
In: Proceedings of the Third Workshop on Automated Semantic Analysis of Information reasoning, which for generations has been cherished as the very
in Legal Text (ASAIL 2019), June 21, 2019, Montreal, QC, Canada. perfection of human reasoning, cannot serve as a model for what
© 2019 Copyright held by the owner/author(s). Copying permitted for private and judges, or indeed what anyone, should do.’
academic purposes.
Published at http://ceur-ws.org These problems are well-known and have been alluded to in the
AI & Law community: ‘The law is normally represented in natural,
ASAIL 2019, June 21, 2019, Montreal, QC, Canada Ilaria Angela Amantea, Luigi Di Caro, Llio Humphreys, Rohan Nanda and Emilio Sulis
albeit technical, language: the language of statutes and cases. These within the international community. There are actors that do not
sources of law are not the law itself, but one possible representation truly have an international legal personality, but are involved and
of the law. It is clear that these documents are not themselves the participate in such activities, including NGOs and multinational
law from the fact, that we must first interpret statutes and cases companies. The first represents public interests of the universal
to get at the law which they represent, and from the fact that civil society, while the second represents productive interests of
reasonable persons can disagree as to just what the law is, although the current economic-financial system [10].
there is rarely disagreement as to what, words make up the statute International law leaves great liberty to states in their choice
or case in question. It is the meaning of the statute or case which is of implementation, being interested only in that the objectives are
the law, not the text of the document itself.’ [17], page 2. achieved. This principle is not dissimilar to treaties of the European
The degree of interpretation required is dependent on the na- Union. Notwithstanding much similarity to international law, Euro-
ture of the legal document itself. Some technical legal instruments, pean Union law has certain characteristics that render it unique. In
such as medical clinical guidelines, are very precise, such that it is particular: "[t]o exercise the Union’s competences, the institutions
possible to identify the hierarchy of norms and model them with shall adopt regulations, directives, decisions, recommendations and
defeasible logic [37]. We argue that a different approach is required opinions. A regulation shall have general application. It shall be
for legislation such as EU directives, which are more contextual and binding in its entirety and directly applicable in all Member States.
abstract. One issue, as mentioned above, is that the determination A directive shall be binding, as to the result to be achieved, upon
of whether a normative provision applies to a particular case is each Member State to which it is addressed, but shall leave to the na-
rendered difficult when the norm is less precise. The other issue is tional authorities the choice of form and methods. A decision shall
that the handling of conflicting legal principles is different to the be binding in its entirety. A decision which specifies those to whom
handling of normative rules. Rather than one principle winning it is addressed shall be binding only on them. Recommendations
to the exclusion of the other, more often conflicting principles are and opinions shall have no binding force"1 .
‘balanced’ to ensure that aspects of both principles are respected The limits of EU competences are governed by the principles
in a way that is proportional and fair. The demarcation between of conferral2 : "the Union shall act only within the limits of the
principles and rules is a point of contention, with some experts competences conferred upon it by the Member States in the Treaties
arguing that it is a continuous spectrum [14]. to attain the objectives set out therein. Competences not conferred
upon the Union in the Treaties remain with the Member States"3 .
The use of Union competences is governed by the principles of:
2.2 The Nature of European Law in the Global
Context • subsidiarity: "in areas which do not fall within its exclu-
sive competence, the Union shall act only if and in so far as
The phenomenon of globalisation has put into question the inter-
the objectives of the proposed action cannot be sufficiently
state structure of the international community. The global dimen-
achieved by the Member States [...] but can rather [...] be bet-
sion of some phenomena (for example climate change, underde-
ter achieved at Union level"4 . Moreover, "when the Treaties
velopment, immigration, terrorism, nuclear proliferation, financial
confer on the Union a competence shared with the Member
transfers, information technology, human rights etc) have shown
States in a specific area, the Union and the Member States
the inadequacy of an international community founded upon the
may legislate and adopt legally binding acts in that area.
independence and sovereignty of states. They require international
The Member States shall exercise their competence to the
rules and institutions due to their transnational character. Tradi-
extent that the Union has not exercised its competence. The
tionally, international law governs relations between independent
Member States shall again exercise their competence to the
states, The norms that bind states originate from their free will in
extent that the Union has decided to cease exercising its
treaties or custom. The development of international law tends to
competence"5 .
reduce state autonomy. In certain matters, such as human rights,
• proportionality: "the content and form of Union action shall
environmental protection, financial crime and terrorism, interna-
not exceed what is necessary to achieve the objectives of the
tional law binds national law. There has been a movement from
Treaties"6 .
international commercial relations from direct inter-state mech-
anisms (above all the General Agreement on Tariffs and Trade) The primary characteristic that distinguishes the European Union
towards a system of international economic relations in the sphere from other international organisations is that member states have
of specific international organisations bestowed with normative, relinquished some sovereign powers to the European Union.
penal and judicial powers. For example, the evolution of GATT It follows other interesting phenomena. First, the European
into the World Trade Organisation, as well as other international Union has the competence to conclude agreements with third states
institutions such as the World Bank created to resolve controversies
in the world of investment. Unlike states, international organisa-
1 Article 288 Treaty on the Functioning of the European Union (TFEU)
tions are not given general competence, but are governed by the 2 Article 5.1 Treaty on European Union (TEU).
principle of specialisation and limited powers to pursue common 3 Underline bot in the Article 4.1 and in the Article 5.2 of TEU.
interests attributed by the states. Not all organisations have a legal 4 Articles 5.1 and 5.3 of TEU. See also Article 4.1 of TEU and Article 352 of TFEU
5 Article 2.2 TFEU. We want to underline that the concept of subsidiarity also exist
personality. To this end, they must a) be given sufficient autonomy,
in some Member States government, but with a meaning more similar to division of
also organisational, distinct from that of member states b) have a competences e.g. see Article 118 of the Italian Constitution.
well-defined mission with corresponding competence and status 6 Articles 5.1 and 5.4 of TEU.
Modelling Norm Types and their Inter-relationships in EU Directives ASAIL 2019, June 21, 2019, Montreal, QC, Canada
or international organisations, where such competences have been We are cognizant that the ECJ has assumed both positions 3 and
expressly given, or if they are acting within their limits. 4 in its judicature in cases 24/6212 and C-162/9713 The proportion
Secondly, according to the "principle of sincere cooperation, the of recitals in directives has increased over the years. Kierkegaard
Union and the Member States shall [...] take any appropriate mea- alleges [23] that “recitals are used by the Member States to insert
sure, general or particular, to ensure fulfilment of the obligations normative provisions which they have failed to get into the text,
arising out of the Treaties or resulting from the acts of the institu- and by the Commission to dump normative provisions which they
tions of the Union [...] and refrain from any measure which could do not want to prolong debate and disagreement on”. From all this
jeopardise the attainment of the Union’s objectives"7 . The increas- we can conclude that recitals cannot be ignored, but their influence
ing trend towards harmonisation of EU law has put the European is uncertain.
Court of Justice as the supreme arbiter of European law, and na-
tional judges are required to interpret their own laws in accordance 3 CLASSIFICATION OF NORM TYPES
with European law, even when there is apparent conflict between There are different kinds of norms with different functions (they
those laws8 . The European Court can "review the legality of acts of serve different purposes), and these different types are found in
bodies, offices or agencies. It shall for this purpose have jurisdiction both normative provisions and recitals. Our categorisation is purely
in actions brought by a Member State, the European Parliament, semantic and is intended to be generalizable to all directives. In
the Council or the Commission on grounds of lack of competence, particular, we have found that the structural aspects (e.g. type of
infringement of an essential procedural requirement, infringement modal verb used) are not a definitive indication. In this work, we
of the Treaties or of any rule of law relating to their application, or have found 5 different types of norms: objective, constitutive, de-
misuse of powers."9 . Furthermore, another particular characteristic ontic, scope and meta-norms (procedural and contextual). All the
of the EU is that "[a]ny natural or legal person may [...] institute examples provided come from a lengthy directive which we have
proceedings against an act addressed to that person or which is of analysed in some detail: Directive 2004/23/EC of the European Par-
direct and individual concern to them", and more important "and liament and of the Council of 31 March 2004 on setting standards
against a regulatory act which is of direct concern to them and does of quality and safety for the donation, procurement, testing, pro-
not entail implementing measures"10 . "If the action is well founded, cessing, preservation, storage and distribution of human tissues
the Court of Justice of the European Union shall declare the act and cells.
concerned to be void"11 . It means that the Court has the power
to "delete" some national law, and with it, delete also the future, Objective
present, and even past effects that law, as if that law never existed.
Definition:
2.3 The Nature of European Directives Outlines the purpose behind the directive as a whole,
or some parts of it, and the wider social and legal
Our focus in this paper is on European directives, which as men-
context. Sometimes the objective is lofty and is a gen-
tioned above, are prescriptive but sufficiently general to allow mem-
eral principle for the existence of the directive. Other
ber states to articulate their own detailed norms and procedures as
times the objective may be a more specific sub-goal.
they prefer in order to achieve the goal of the directive. Being by
nature goal-oriented, directives are particularly given to principle- Example 1:
based (balance) rather than defeasible reasoning. Almost half of ARTICLE 1: This Directive lays down standards of
the text of directives consists of recitals, which are intended to be quality and safety for human tissues and cells in-
explanatory and do not have the same status as normative provi- tended for human applications, in order to ensure
sions. There are different doctrinal positions [24] on the relationship a high level of protection of human health.
between recitals and normative provisions:
Constitutive
(1) recitals have no effect;
(2) recitals are dominant over normative provisions; Definition:
(3) recitals have an equal position in relation to normative pro- Official definitions of directive-specific technical con-
visions; cepts. Legislative constitutive norms are usually gen-
(4) recitals encompass a subordinate position towards normative eral descriptions, but also not uncommon are defini-
provisions. tions by example, which allow extension by analogy,
as well as definitions that explicitly include or exclude
certain items from counting as the legal concept in
question.
12 Position 3: Case 24/62, F.R.G. v. Comm’n of the Eur. Econ. Cmty., 1963 E.C.R., para-
7 Article 4.3 of TEU.
graph 18, retrieved at
8 See Articles 258 to 260 of TFEU.
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:61962J0024:EN:NOT
9 Article 263, pararaphs 1 and 2 of TFEU. 13 Position 4: Case C-162/97, Nilsson et al, paragraph 54, 1998, E.C.R. I-07477, available
10 Article 263, paragraph 4 of TFEU.
at
11 Article 264 of TFEU http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:61997J0162:EN:NOT
ASAIL 2019, June 21, 2019, Montreal, QC, Canada Ilaria Angela Amantea, Luigi Di Caro, Llio Humphreys, Rohan Nanda and Emilio Sulis
Example 2: Procedural refers to step-by-step processes for imple-
menting law e.g. get signatures, agreement from the
ARTICLE 3: For the purposes of this Directive: (a)
Committee, further signatures.
“cells” means individual human cells or a collection of
human cells when not bound by any form of connec- Example 5:
tive tissue; (b) [...] ARTICLE 10.3: Member States and the Commission
shall establish a network linking the national tissue
Deontic establishment registers.
Definition: While the above example is a procedure particular to the subject-
matter of the directive, there are also other procedural norms that
Specifies types of behaviour to be expected or per- occur in practically all directives, and are part of the law-making
mitted. Deontic norms have been further classified as process.
permission, obligation, prohibition etc (see [21] for an
elaborate study), but this level of detail is not required Example 6:
for our purposes. ARTICLE 26.1: Member States shall send the Com-
Example 3: mission, before 7 April 2009 and every three years
thereafter, a report on the activities undertaken in
ARTICLE 7.7: Member States shall, upon the request relation to the provisions of this Directive, includ-
of another Member State or the Commission, provide ing an account of the measures taken in relation to
information on the results of inspections and control inspection and control.
measures carried out in relation to the requirements
of this Directive. Meta-norms: Contextual
Definition:
Scope
Contextualisation is about time, space, addressee and
Definition: hierarchy of norms.
Example 7:
Outlines the extent of applicability or non-applicability
of norms (or entire legislation) in the context of other ARTICLE 32: This Directive shall enter into force on
norms (or other legislation) with which they may oth- the day of its publication in the Official Journal of the
erwise conflict. Scope also concerns norms that spec- European Union.
ify the areas of competence in different jurisdictions,
in this case the EU and member states. 4 CLASSIFICATION OF LINK TYPES
Example 4: Norms are related to one another in many different ways. Below
we provide definitions with examples from the above-mentioned
RECITAL 11: This Directive does not cover research Directive 2004/23/EC, and are mainly focused on links between
using human tissues and cells, such as when used for normative provisions and recitals. Where not all parts of the recital
purposes other than application to the human body, or (sub)article are connected, the underlined sections highlight the
e.g. in vitro research or in animal models. Only those related parts of the text.
cells and tissues that in clinical trials are applied to
the human body should comply with the quality and Conceptually Similar
safety standards laid down in this Directive.
Definition:
Meta-norms There is content within the provisions that are about
the same subject-matter and may use similar or dif-
Meta-norms are norms about norms. Laws do not only pertain ferent wording to say more or less the same thing.
to certain behaviours but also (1) the way in which the norms Example 8 (using same wording):
are produced, (2) the way in which they are applied and (3) the
way in which upon violation a sanction is imposed. The distinction ARTICLE 2.1: This Directive shall apply to the donation,
between norms and meta-norms is that the first concerns behaviour, procurement, testing, processing, preservation, storage
the second concerns other norms or the production or application and distribution of human tissues and cells intended
of other norms [15]. We distinguish between two kinds of meta- for human applications and of manufactured prod-
norms: ucts derived from human tissues and cells intended
for human applications. Where such manufactured
Meta-norms: Procedural products are covered by other directives, this Direc-
tive shall apply only to donation, procurement and
Definition: testing.
Modelling Norm Types and their Inter-relationships in EU Directives ASAIL 2019, June 21, 2019, Montreal, QC, Canada
RECITAL 13: The donation, procurement, testing, body, so that it is as similar as possible to its original
processing, preservation, storage and distribution of anatomical shape.
human tissues and cells intended for human applications
In this case Article 3 contains definitions of many domain-specific
should comply with high standards of quality and
terms in Recital 16, so there is a Constitutive link between these
safety in order to ensure a high level of health protec-
provisions.
tion in the Community. This Directive should estab-
lish standards for each one of the steps in the human Motivation
tissues and cells application process.
Definition:
Example 9 (using different wording):
A link between a deontic norm and the motivation
ARTICLE 5.1: Member States shall ensure that tissue and behind it. The provisions have the same goal but dif-
cell procurement and testing are carried out by persons ferent levels of granularity. Sometimes the motivation
with appropriate training and experience and that they is lofty and is a general principle for the existence of
take place in conditions accredited, designated, autho- the directive, alternatively the motivation may be a
rised or licensed for that purpose by the competent core value of human rights or a fundamental principle
authority or authorities. of European Treaties. Other times the motivation may
RECITAL 27: Personnel directly involved in the dona- be a more specific sub-goal.
tion, procurement, testing, processing, preservation, Example 11:
storage and distribution of human tissues and cells ARTICLE 16.3: Tissue establishments shall take all
should be appropriately qualified and provided with necessary measures to ensure that the quality system
timely and relevant training. The provisions laid down includes at least the following documentation:
in this Directive as regards training should be applica- – standard operating procedures,
ble without prejudice to existing Community legisla- – guidelines,
tion on the recognition of professional qualifications. – training and reference manuals,
– reporting forms,
Constitutive – donor records,
– information on the final destination of tissues or
Definition: cells.
The constitutive link is where one provision provides RECITAL 1: The transplantation of human tissues and
a definition of domain-specific terms contained in cells is a strongly expanding field of medicine offering
another provision. Often, directives contain a glossary great opportunities for the treatment of as yet incur-
of terms in one specific Article. able diseases. The quality and safety of these substances
Example 10: should be ensured, particularly in order to prevent the
transmission of diseases.
ARTICLE 3: For the purposes of this Directive:
(a) ‘cells’ means individual human cells or a collection RECITAL 15: It is necessary to increase confidence
of human cells when not bound by any form of con- among the Member States in the quality and safety of
nective tissue; donated tissues and cells, in the health protection of
(b) ‘tissue’ means all constituent parts of the human living donors and respect for deceased donors and in
body formed by cells; the safety of the application process.
(c) ‘donor’ means every human source, whether living
or deceased, of human cells or tissues; Impact
(d) ‘donation’ means donating human tissues or cells
intended for human applications; [...] Definition:
(f) ‘procurement’ means a process by which tissue or A provision may affect the scope or effectiveness of
cells are made available; [...] another provision, or add additional requirements.
(p) ‘allogeneic use’ means cells or tissues removed This may be because the norm requires some kind of
from one person and applied to another; [...] synchronisation with another procedure in another
norm, or because two norms have conflicting goals.
RECITAL 16 Tissues and cells used for allogeneic ther-
The content may be granularity independent.
apeutic purposes can be procured from both living
and deceased donors. In order to ensure that the health Example 12 (conflicting goals):
status of a living donor is not affected by the donation, ARTICLE 8.1: Member States shall ensure that all tis-
a prior medical examination should be required. The sues and cells procured, processed, stored or distributed
dignity of the deceased donor should be respected, on their territory can be traced from the donor to the
notably through the reconstruction of the donor’s recipient and vice versa. This traceability shall also apply
ASAIL 2019, June 21, 2019, Montreal, QC, Canada Ilaria Angela Amantea, Luigi Di Caro, Llio Humphreys, Rohan Nanda and Emilio Sulis
to all relevant data relating to products and materials transmit information about serious adverse events and
coming into contact with these tissues and cells. reactions which may influence the quality and safety of
RECITAL 23: All necessary measures need to be taken tissues and cells and which may be attributed to the
in order to provide prospective donors of tissues and procurement, testing, processing, storage and
cells with assurances regarding the confidentiality of any distribution of tissues and cells, as well as any serious
health related information provided to the authorised adverse reaction observed during or after clinical
personnel, the results of tests on their donations, as application which may be linked to the quality and
well as any future traceability of their donation. safety of tissues and cells.
RECITAL 25: An accreditation system for tissue estab-
We can reason that traceability is a sub-goal of health protection, lishments and a system for notification of adverse events and
and that confidential information is a sub-goal of respecting the
reactions linked to the procurement, testing, processing,
dignity of the individual, and that both these higher goals are sub-
preservation, storage and distribution of human tissues and
goals of maintaining the well-being of persons. However, there is a
cells should be established in the Member States.
potential risk of conflict between these sub-goals e.g. traceability
can be achieved without anonymity. As such, it is useful to link In this example, Article 11.3 is indirectly structurally linked to
such norms to any norms that impose restrictions or additional Recital 25 because Article 11.3 refers to Article 11.1, which is Con-
requirements. ceptually Similar to Recital 25.
Example 13 (enforcement): Via Other Law
ARTICLE 6.3: The tissue establishment shall not under-
take any substantial changes to its activities without Definition:
the prior written approval of the competent authority
When an article mentions another legal source. Ap-
or authorities.
plies only if reading this source is required in order to
RECITAL 26: Member States should organise inspections understand the provision or recital. This is a structural
and control measures, to be carried out by officials link, and not a semantic link.
representing the competent authority, to ensure that Example 15:
tissue establishments comply with the provisions of this
ARTICLE 13.1 The procurement of human tissues or
Directive. Member States should ensure that the offi-
cells shall be authorised only after all mandatory con-
cials involved in inspections and control measures are
sent or authorisation requirements in force in the
appropriately qualified and receive adequate training.
Member State concerned have been met.
RECITAL 30: In order to increase the effective imple-
RECITAL 22: This Directive respects the fundamen-
mentation of the provisions adopted in accordance
tal rights and observes the principles reflected in the
with this Directive, it is appropriate to provide for
Charter of Fundamental Rights of the European Union
penalties to be applied by Member States.
(1) and takes into account as appropriate the Conven-
The impact of Recitals 26 and 30 on Article 6.3 is to render the norm tion for the protection of human rights and dignity
enforceable via monitoring and penalties. of the human being with regard to the application of
biology and medicine: Convention on human rights
Indirect Internal and biomedicine. Neither the Charter nor the Con-
vention makes express provision for harmonisation
Definition: or prevents Member States from introducing more
An indirect internal link is a structural (not semantic) stringent requirements in their legislation.
link that exists purely because of an internal reference
to a (sub)article. An indirect structural link between In the above example, the fundamental rights and dignity of the
Recital X and (Sub)article Y depends on the existence human being mentioned in Recital 22 is the motivation behind the
of a primary semantic link between Recital X and requirement for consent for the procurement of human tissues and
another (sub)article that is cited by (Sub)article Y. cells in Article 13.1.
Example 14: Procedural
ARTICLE 11.3: The responsible person referred to in
Article 17 shall ensure that the competent authority or Definition:
authorities is or are notified of any serious adverse events Involving routine bureaucratic procedures of some
and reactions referred to in paragraph 1 and is or are precision. The procedures referred to here are only
provided with a report analysing the cause and the descriptions of what the Commission, other EU body,
ensuing outcome. or Member State, have undertaken to do to render the
ARTICLE 11.1: Member States shall ensure that there directive effective. They are not norms in the deontic
is a system in place to report, investigate, register and sense. Procedural links do not pertain to norms about
Modelling Norm Types and their Inter-relationships in EU Directives ASAIL 2019, June 21, 2019, Montreal, QC, Canada
what lower bodies within Member States are required affect quality and safety and shall ensure that they
to do, which are still deontic norms, in the sense that are carried out under controlled conditions. Tissue
they impose an obligation. establishments shall ensure that the equipment used,
Example 16: the working environment and process design, valida-
ARTICLE 29.1: The Commission shall be assisted by a tion and control conditions are in compliance with
Committee. the requirements referred to in Article 28(h).
RECITAL 34: The measures necessary for the imple- ARTICLE 20.3: Tissue establishments shall include
mentation of this Directive should be adopted in accor- in their standard operating procedures special provi-
dance with Council Decision 1999/468/EC of 28 June sions for the handling of tissues and cells to be dis-
1999 laying down the procedures for the exercise of carded, in order to prevent the contamination of other
implementing powers conferred on the Commission. tissues or cells, the processing environment or per-
sonnel.
The procedural link exists to distinguish between deontic norms ARTICLE 21.1: Tissue establishments shall ensure that
and descriptions of procedures that render the law effective. How- all procedures associated with the storage of tissues
ever, the linked sub-articles and recitals can have different levels of and cells are documented in the standard operating
granularity. procedures and that the storage conditions comply
Contextual with the requirements referred to in Article 28(h).
Definition:
5 DISCUSSION
Contextualising the applicability of all norms involved Our classification of norm types and link types was data-driven and
in the directive in terms of time, jurisdiction, addressee based on line-by-line coding with an open mind, as espoused by
and position in the hierarchy of norms. grounded theory[11, 12], followed by comparison and alignment
The following are the contextual meta-norms that with theories in the literature. The analysis was carried out by
should be linked to other articles and recitals of Di- researchers with a background in law, computer science and legal
rective 2004/23/EC. informatics. While the categories below were agreed by all to be
evident in the data, the attribution of individual provisions to a
Example 17:
particular category was more problematic. As such, our future work
ARTICLE 32 (time): This Directive shall enter into will involve annotation of a corpus of directives by a larger group of
force on the day of its publication in the Official Jour- annotators in order to properly elicit the distribution of categories,
nal of the European Union. evaluate inter-annotation agreement and, where necessary, refine
ARTICLE 33 (addressees): This Directive is addressed the categories. In this preliminary work, we articulate below some
to the Member States. of the theoretical questions that require further analysis.
RECITAL 31 (jurisdiction): Since the objective of this In this work, we have used the term deontic norm to describe
Directive, namely to set high standards of quality and what we originally called detailed technical norms and to distin-
safety for human tissues and cells throughout the guish them from objectives. In some legal traditions, e.g. Italian,
Community, cannot be sufficiently achieved by the detailed norms also may include norms that are not strictly deontic.
Member States and can therefore, by reason of scale Further work is required to explore the correct classification of
and effects, be better achieved at Community level, these norms.
the Community may adopt measures in accordance The observant reader will have noticed that granularity is treated
with the principle of subsidiarity as set out in Article in different ways for different classes. Relevant sections in Concep-
5 of the Treaty. In accordance with the principle of tually Similar links are on the same level of granularity. On the
proportionality, as set out in that Article, this Direc- other hand, a Motivation link features different levels of granu-
tive does not go beyond what is necessary in order to larity, although the difference in granularity may be big or small.
achieve that objective. For Impact, Procedural and Contextual, the level of granularity is
inconsequential. Is this difference justifiable? Should there be sub-
Norm Group categories beyond the classes identified in this work e.g. a ’general
principle’ versus ’immediate goal’ type of Motivation? Or does the
Definition: granularity issue indicate that there is something fundamentally
A link between norms that are connected due to be- problematic about the classes as they are? More work is required
ing part of the same general requirement. The links to explore this issue.
between the norms may be conjunction, disjunction
or sequence. Such norms may be paragraphs of the 6 INSIGHTS FOR AUTOMATED
same article, or may occur in different provisions. IDENTIFICATION
Example 18: The automated identification of norm types is a task which lends
ARTICLE 20.1: Tissue establishments shall include in itself to well-known supervised classification techniques, such as
their standard operating procedures all processes that SVM, Bayes, decision trees, and most recently neural networks [26].
ASAIL 2019, June 21, 2019, Montreal, QC, Canada Ilaria Angela Amantea, Luigi Di Caro, Llio Humphreys, Rohan Nanda and Emilio Sulis
The automated classification of links types, on the other hand, Dutch legislation respectively. In the first paper, provisions are
requires closer scrutiny. The input is two different texts and the classified as definitions, attributing competence, constitutive, inter-
output is the relation between them. Much literature on relevance, pretative, instituting, prescriptive, procedural, sanctioning, material
particularly in information retrieval, assume that relevance corre- link (derogation or extension, amending link (abrogation, substitu-
lates with similarity. However, our analysis shows that norms are tion), temporal link (prorogation, suspension). The second paper
related in different ways. Our intuition is that different techniques finds core rules, procedures for citizens, procedures for civil ser-
are required to identify those links, and we intend to experiment vants, rule management and definitions in the body of law, with
with lexical-syntactic, knowledge-based and corpus-based tech- introduction, conclusion and appendices completing the model of
niques. Lexical-syntactic techniques are based on the words in the legislative text. Although there are similarities in the categorisation,
input data. They include edit distance [8], pattern-based [5], N- there are also differences due to the nature of the legislation being
grams and Longest Common Subsequence [25]. Thesaurus-based modelled. EU law is particular for having less definite, more goal-
approaches [31, 38] rely on background knowledge. Corpus-based based and principle-based norms both in the recitals and provisions.
techniques include word embeddings, such as Word2Vec [32] or We would point out a couple of key differences in the classification.
GloVe [36], and topic modeling [30, 50]. We envisage the follow- Tiscornia and Turchi [46] define constitutive norms more broadly
ing techniques for each link type: word embeddings and topic than us to include also power-conferring norms. De Maat [13]’s
modelling for Conceptually Similar links (since the similar texts categorisation includes procedural norms also for the addressees
need not necessarily contain the same wordings), pattern-based of norms, whereas our procedural norms describe the procedures
approaches for Constitutive links, thesaurus-based approaches for undertaken by institutions of the EU to implement and maintain EU
Motivation links (in order to capture information of different speci- legislation. Another important factor is in the motivation for clas-
ficity), topic modeling for Impact, pattern-based for Indirect Internal sifying the norms. The classification of Tiscornia and Turchi [46]
links, pattern-based for Via Other Law links, pattern-based for Pro- and De Maat [13] serve to model the content of norms. Our classifi-
cedural links in combination with a set of trigger words, n-grams cation of norms may help in determining the relatedness between
and longest common subsequence for Contextual links, and word norms.
embeddings for Norm Group links. The automated classification of norms have also been explored
We believe that an eventual support system should take a greedy by Biagioli et al [16] and Waltl et al [51, 53]. The first used Sup-
approach to selecting a low threshold for relevance. A high recall, port Vector Machines, while the second found similar performance
low precision system means that users have to put up with the pre- by local linear approximations and a manually crafted rule-based
sentation of irrelevant data, but this is still less work than manually approach.
looking at all possibilities. On the other hand, a high precision, low Important theoretical analysis on contextualisation has been
recall system, means that users may be confident in the validity conducted in the development of annotation standards for mod-
of the classification, but then have to go through all possibilities elling legislation. The temporal dimension is modelled in Legal
to find others missed by the system. While a greedy approach is, Knowledge Interchange Format (LKIF) [35] by ascribing to each
generally speaking, the most pragmatic approach to designing user norm blocks of information covering time of entry into force, time
support systems, there is a danger of information overload ren- of efficacy and time of application. Akoma Ntoso [6] has a multi-
dering the support system frustrating to use. The classic solution layered approach to modelling laws with the text layer continuing
to this problem is to rank the results. However, the issue of how the original legal text, the structure layer providing a hierarchi-
to rank the links is not simple for this work. Sections which are cal representation of the parts present in the text layer, and the
identical in meaning do not offer new information, and therefore metadata layer associating the first two layers with ontological
may not be the most valuable to a legal researcher. Then, there are information to allow advanced reasoning using logic frameworks,
recitals providing general principles that motivate various norms. and allowing for multiple interpretations of norms by different ac-
Are these so generic that they can be safely ignored? Only per- tors. Akoma Ntoso also incorporates the Functional Requirements
haps to those who are already familiar with the directive under for Bibliographic Record (FRBR) model [42] to identify different
consideration. Otherwise, even general principles can on occasion versions of the same work. The importance of contextualisation is
help disambiguate unclear provisions if one plausible interpretation well-explained in [4], page 152: ’[P]rovisions, rules, applications of
would clearly go against the purpose of the entire directive. Perhaps rules, references to text, and references to physical entities. All of
the key to creating a user-friendly system is careful interface design these entities exist and change in time; their histories interact in
that enables users to select the information they want. For instance, complicated ways...[A] rule has parameters which can vary over
they could filter out certain classes of links. Or they could choose time, such as its status (e.g., strict, defeasible, defeater), its validity
whether to e.g. display Motivation links in order of Most Specific or (e.g., repealed, annulled, suspended), and its jurisdiction (e.g., only
Most General, Conceptually Similar links in order of Most Similar in EU, only in US). In addition, a rule has temporal aspects such as
or Most Dissimilar etc. internal constituency of the action, the time of assertion of the rule,
the efficacy, enforcement, and so on.’ As such LegalRuleML anno-
tates each rule with its defeasibility status, temporality, jurisdiction
7 RELATED WORK and authorial tracking.
On the classification of norms, the papers of Tiscornia and Turchi [46], On the classification of links between norms, we refer again to
and de Maat [13] are notable for developing their classification Tiscornia and Turchi [46], who also define links between norms.
informed by legal philosophy, while being tested on Italian and Their links are by logical implication - a norm of one class, such as
Modelling Norm Types and their Inter-relationships in EU Directives ASAIL 2019, June 21, 2019, Montreal, QC, Canada
prohibition, will of necessity be linked to a norm of another class, importance of digital information for legal professionals - lawyers
such as sanction. Our classification of link types are generally more easily spend up to fifteen hours per week on search, most of it in
semantic and probabilistic. electronic resources although the abandonment of paper does not
An important work on normative similarity is that of Lau [29]. always seem to be a voluntary choice - the gap between LIR systems
Provisions are tagged with the stems of noun phrases, legislative def- and user needs is still big’. This is because ’retrieval engineering is
initions or glossaries from reference books. The similarity analysis focused too exclusively on algorithmic relevance, but it has been
core takes as an input the parsed regulations and associated fea- proven sufficiently that without domain specific adaptations every
tures, and produces a list of the most similar pairs of provisions. The search engine will disappoint legal users’ (ibid, page 84). While there
most similar research to this paper in terms of its subject-matter is are several projects that seek to find linked legal data from different
that of Humphreys et al [22], which explored the feasibility of semi- sources, such as EUCases [9] and the LATC project (Linked Open
automated mapping between normative provisions and recitals. Data Around-The-Clock)15 , in this article we chiefly looked at the
However, the focus of that paper was deontic norms and the links relationship between norms within the same piece of legislation.
explored were only Conceptually Similar. The work of Nanda et
al [34] is concerned with finding norms from national legislation 8 CONCLUSIONS AND FUTURE WORK
that implement norms from EU directives. This thorough research This paper sought to model norm types and their inter-relationships
investigates a variety of similarity algorithms suited for short text, in EU directives. Recitals and provisions were categorised and the
including a unifying text similarity measure (USM) which incorpo- links between them were also categorised. We started from the
rates methods for matching common words, common sequences of presumption of the hyper-textuality and high-level nature of EU
words and approximate string matching. While informative for our directives, which are deliberately imprecise to allow Member States
purposes, research on automated mechanisms for finding similar to fulfil the objectives in their own way. It follows that deontic
norms is only part of our agenda, since we seek to extend the notion norms in EU directives cannot be properly understood without
of relatedness, and will require different approaches for different consideration of the motivation behind those norms, as well as con-
types of links. sideration of how other provisions, or even other laws, may affect
On the nature of relations between norms, there is a wealth of the scope or effectiveness of the norm in question. Our analysis
theoretical research on conflicts between norms [2] and defeasibility includes the impact of meta-norms that render the norms effective
(summarised in [37]). The main features of formal approaches to and determine when the provisions are valid and in which context
defeasibility are: (time, space, jurisdiction).
• arguments that are satisfied with certain criteria; The nature of EU directives means that modelling the inter-
• counter-arguments that serve to attack or undercut other relationship between norms with defeasible logic cannot be un-
arguments; dertaken with certainty. This does not mean AI & Law cannot be
• the non-monotonic nature of legal reasoning, where conclu- useful. Rather, we propose a different approach: to find automated
sions can be revised with the addition of new information. means to find norms that are related in different ways in order to
Such analyses can be used to build a formal legal reasoning expert help legal professionals interpret laws for specific cases.
system such as that of Sergot et al [44] and Lam & Governatori [27]. Our future work will involve the creation of a gold standard to
Some attention has been paid to contrary-to-duty obligations i.e. a help build a system to semi-automatically classify norms and their
conditional obligation arising in response to a violation of another inter-relationships. Legal corpora have been created for a variety of
obligation, particularly for compliance management [18]. The use of different purposes including classifying norm types and extracting
automated systems in legal settings has gained credibility with the norm elements ([7] and [49]), ontology learning (e.g. [45]) and
rise of AI in general. Even robotic judges are becoming acceptable question and answering [33]. There is no specific legal corpora
for limited domains in Estonia14 . However, there are certain kinds on norm types beyond deontic norms and the linking of related
of laws that are simply too abstract or dependent on other laws to provisions as far as we know.
be modelled in this way with certainty. Our preliminary analysis suggests that different algorithms are
There is also a wealth of literature on the discovery and classifi- required to identify different types of norms and their inter-
cation of citations in legislation[1, 3, 28, 40, 52]. This work is chiefly relationships. Our future work will involve experimentation to dis-
concerned with structural references between norms of the same cover which algorithms are most appropriate for each case. More-
and different legislation, and we will look closely at the techniques over, we envisage that a semi-automated classification of norm
used with respect to Indirect Structural and Via Other Law links. types and links may enable the end-user in a support system to
Interestingly, in addition to full-explicit references, semi-explicit select which kind of norms and links (s)he wishes to view, further
references, and implicit references, Waltl et al [52] also refer to reducing the problem of information overload.
tacit references which are defined as follows: “[t]he connection
between the norms emerges due to systemic interpretation and REFERENCES
cannot not be determined by exclusively analyzing the norm text.” [1] Morayo Adedjouma, Mehrdad Sabetzadeh, and Lionel C Briand. 2014. Automated
Tacit references are not covered by their system. detection and resolution of legal cross references: Approach and a study of lux-
embourg’s legislation. In 2014 IEEE 22nd International Requirements Engineering
The case for support systems to aid legal support was made Conference (RE). IEEE, 63–72.
by Opijnen and Santos [48], page 84, which states that ‘given the
14 ://www.wired.com/story/can-ai-be-fair-judge-court-estonia-thinks-so/ 15 http://latc-project.eu
ASAIL 2019, June 21, 2019, Montreal, QC, Canada Ilaria Angela Amantea, Luigi Di Caro, Llio Humphreys, Rohan Nanda and Emilio Sulis
[2] Carlos E Alchourrón and David Makinson. 1981. Hierarchies of regulations and [30] Jon D Mcauliffe and David M Blei. 2008. Supervised topic models. In Advances in
their logic. In New studies in deontic logic. Springer, 125–148. neural information processing systems. 121–128.
[3] Kevin D Ashley, Matthias Grabmair, and Rebecca Hwa. [n. d.]. Network Analysis [31] Rada Mihalcea, Courtney Corley, Carlo Strapparava, et al. 2006. Corpus-based and
of Manually–Encoded State Laws and Prospects for Automation. In Workshop knowledge-based measures of text semantic similarity. In AAAI, Vol. 6. 775–780.
Format. 1. [32] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.
[4] Tara Athan, Guido Governatori, Monica Palmirani, Adrian Paschke, and Adam Z. Distributed representations of words and phrases and their compositionality. In
Wyner. 2015. LegalRuleML: Design Principles and Foundations. In Reasoning Advances in neural information processing systems. 3111–3119.
Web. Web Logic Rules - 11th International Summer School 2015, Berlin, Germany, [33] Alfredo Monroy, Hiram Calvo, and Alexander Gelbukh. 2009. NLP for shallow
July 31 - August 4, 2015, Tutorial Lectures. 151–188. https://doi.org/10.1007/ question answering of legal documents using graphs. In International Conference
978-3-319-21768-0_6 on Intelligent Text Processing and Computational Linguistics. Springer, 498–508.
[5] Alain Auger and Caroline Barrière. 2008. Pattern-based approaches to semantic [34] Rohan Nanda, Luigi Di Caro, Guido Boella, Hristo Konstantinov, Tenyo Tyankov,
relation extraction: A state-of-the-art. Terminology 14, 1 (2008), 1–19. Daniel Traykov, Hristo Hristov, Francesco Costamagna, Llio Humphreys, Livio
[6] Gioele Barabucci, Luca Cervone, Monica Palmirani, Silvio Peroni, and Fabio Robaldo, and Michele Romano. 2017. A unifying similarity measure for auto-
Vitali. 2009. Multi-layer markup and ontological structures in Akoma Ntoso. mated identification of national implementations of european union directives.
In International Workshop on AI Approaches to the Complexity of Legal Systems. In Proceedings of the 16th edition of the International Conference on Articial Intelli-
Springer, 133–149. gence and Law, ICAIL 2017, London, United Kingdom, June 12-16, 2017. 149–158.
[7] Roberto Bartolini, Alessandro Lenci, Simonetta Montemagni, Vito Pirrelli, and https://doi.org/10.1145/3086512.3086527
Claudia Soria. 2004. Automatic classification and analysis of provisions in italian [35] Monica Palmirani, Guido Governatori, and Giuseppe Contissa. 2011. Modelling
legal texts: a case study. In OTM Confederated International Conferences" On the temporal legal rules. In Proceedings of the 13th International Conference on Artificial
Move to Meaningful Internet Systems". Springer, 593–604. Intelligence and Law. ACM, 131–135.
[8] Philip Bille. 2005. A survey on tree edit distance and related problems. Theoretical [36] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove:
computer science 337, 1-3 (2005), 217–239. Global vectors for word representation. In Proceedings of the 2014 conference on
[9] Guido Boella, Luigi Di Caro, Michele Graziadei, Loredana Cupi, Carlo Emilio empirical methods in natural language processing (EMNLP). 1532–1543.
Salaroglio, Llio Humphreys, Hristo Konstantinov, Kornel Marko, Livio Robaldo, [37] Henry Prakken and Giovanni Sartor. 2004. The three faces of defeasibility in the
Claudio Ruffini, et al. 2015. Linking legal open data: breaking the accessibility law. Ratio Juris 17, 1 (2004), 118–139.
and language barrier in european legislation and case law. In Proceedings of the [38] Ray Richardson, A Smeaton, and John Murphy. 1994. Using WordNet as a
15th International Conference on Artificial Intelligence and Law. ACM, 171–175. knowledge base for measuring semantic similarity between words.
[10] Sergio Maria Carbone, Riccardo Luzzatto, Stefania Bariatti, Paola Ivaldi, Ilaria [39] Rodolfo Sacco. 1991. Legal formants: a dynamic approach to comparative law
Queirolo, Francesco Munari, Luigi Fumagalli, Bruno Nascimbene, Lorenzo Schi- (Installment I of II). The American Journal of Comparative Law 39, 1 (1991), 1–34.
ano Di Pepe, Manlio Frigo, et al. 2016. Istituzioni di diritto internazionale. G [40] Ali Sadeghian, Laksshman Sundaram, D Wang, W Hamilton, Karl Branting,
Giappichelli Editore. and Craig Pfeifer. 2016. Semantic edge labeling over legal citation graphs. In
[11] Kathy Charmaz. 1990. âĂŸDiscoveringâĂŹchronic illness: using grounded theory. Proceedings of the workshop on legal text, document, and corpus analytics (LTDCA-
Social science & medicine 30, 11 (1990), 1161–1172. 2016). 70–75.
[12] Juliet M Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, [41] Mark Sainsbury. 2000. Logical forms: An introduction to philosophical logic.
canons, and evaluative criteria. Qualitative sociology 13, 1 (1990), 3–21. (2000).
[13] Emile De Maat and Radboud Winkels. 2007. Categorisation of norms. FRONTIERS [42] KG Saur. 1998. IFLA Study Group on the functional requirements for bibliographic
IN ARTIFICIAL INTELLIGENCE AND APPLICATIONS 165 (2007), 79. records. Functional requirements for bibliographic records: final report.
[14] P Caretti-U De Siervo. 2002. Istituzioni di diritto pubblico. Giappichelli, Torino, [43] J.R. Searle. 1995. The Construction of Social Reality. London: Penguin.
ultima ed.(dal Cap. X alla fine) (2002). [44] Marek J. Sergot, Fariba Sadri, Robert A. Kowalski, Frank Kriwaczek, Peter Ham-
[15] Carla Faralli. 2016. Argomenti di Teoria del Diritto. G Giappichelli Editore. mond, and H Terese Cory. 1986. The British Nationality Act as a logic program.
[16] Enrico Francesconi and Andrea Passerini. 2007. Automatic classification of Commun. ACM 29, 5 (1986), 370–386.
provisions in legislative texts. Artificial Intelligence and Law 15, 1 (2007), 1–17. [45] Peter Spyns and Marie-Laure Reinberger. 2005. Lexically evaluating ontology
[17] Thomas F Gordon. 1986. The Role of Exceptions in Models of the Law. Formal- triples generated automatically from texts. In European Semantic Web Conference.
isierung im Recht und Ansätze juristischer Expertensysteme (1986), 52–59. Springer, 563–577.
[18] Guido Governatori and Antonino Rotolo. 2010. A conceptually rich model of [46] Daniela Tiscornia and Fabrizio Turchi. 1997. Formalization of legislative doc-
business process compliance. In Proceedings of the Seventh Asia-Pacific Conference uments based on a functional model. In International Conference on Artificial
on Conceptual Modelling-Volume 110. Australian Computer Society, Inc., 3–12. Intelligence and Law: Proceedings of the 6 th international conference on Artificial
[19] Davide Grossi, John-Jules Meyer, and Frank Dignum. 2008. The Many Faces of intelligence and law, Vol. 30. Citeseer, 63–71.
Counts-as: A Formal Analysis of Constitutive-rules. Journal of Applied Logic [47] Charles Travis. 1994. On constraints of generality. In Proceedings of the Aristotelian
6, 2 (2008), 192–217. http://www.davidegrossi.name/Site/publications_files/ Society, Vol. 94. JSTOR, 165–188.
grossi07many.pdf [48] Marc Van Opijnen and Cristiana Santos. 2017. On the concept of relevance in
[20] Herbert Lionel Adolphus Hart. 1957. Positivism and the Separation of Law and legal information retrieval. Artificial Intelligence and Law 25, 1 (2017), 65–87.
Morals. Harv. L. Rev. 71 (1957), 593. [49] Giulia Venturi. 2011. Semantic annotation of Italian legal texts: a FrameNet-based
[21] Risto Hilpinen and Paul McNamara. 2013. Deontic logic: A historical survey and approach. Constructions and Frames 3, 1 (2011), 46–79.
introduction. Handbook of deontic logic and normative systems 1 (2013), 3–136. [50] Hanna M Wallach. 2006. Topic modeling: beyond bag-of-words. In Proceedings of
[22] Llio Humphreys, Cristiana Santos, Luigi Di Caro, Guido Boella, Leon Van the 23rd international conference on Machine learning. ACM, 977–984.
Der Torre, and Livio Robaldo. 2015. Mapping Recitals to Normative Provisions [51] Bernhard Waltl, Georg Bonczek, Elena Scepankova, and Florian Matthes. 2019.
in EU Legislation to Assist Legal Interpretation.. In JURIX. 41–49. Semantic types of legal norms in German laws: classification and analysis using
[23] Sylvia Mercado Kierkegaard. 2006. Here comes the âĂŸcybernatorsâĂŹ! Com- local linear explanations. Artificial Intelligence and Law 27, 1 (2019), 43–71.
puter Law & Security Review 22, 5 (2006), 381–391. [52] Bernhard Waltl, Jörg Landthaler, and Florian Matthes. 2016. Differentiation and
[24] T. Klimas and J. Vaiciukaite. 2008. The Law of Recitals in European Community Empirical Analysis of Reference Types in Legal Documents.. In JURIX. 211–214.
Legislation. ILSA Journal of International & Comparative Law 15, 6 (7 2008), 61–93. [53] Bernhard Waltl, Johannes Muhr, Ingo Glaser, Georg Bonczek, Elena Scepankova,
Available at: https://nsuworks.nova.edu/ilsajournal/vol15/iss1/6. and Florian Matthes. 2017. Classifying Legal Norms with Active Machine Learn-
[25] Grzegorz Kondrak. 2005. N-gram similarity and distance. In International sympo- ing.. In JURIX. 11–20.
sium on string processing and information retrieval. Springer, 115–126.
[26] Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu,
Laura Barnes, and Donald Brown. 2019. Text Classification Algorithms: A Survey.
Information 10, 4 (2019), 150.
[27] Ho-Pun Lam and Guido Governatori. 2009. The making of SPINdle. In Inter-
national Workshop on Rules and Rule Markup Languages for the Semantic Web.
Springer, 315–322.
[28] Jörg LANDTHALER, Ingo GLASER, and Florian MATTHES. 2018. Towards
Explainable Semantic Text Matching. In Legal Knowledge and Information Systems:
JURIX 2018: The Thirty-first Annual Conference, Vol. 313. IOS Press, 200.
[29] Gloria T Lau, Kincho H Law, and Gio Wiederhold. 2003. Similarity analysis on
government regulations. In Proceedings of the ninth ACM SIGKDD international
conference on Knowledge discovery and data mining. ACM, 711–716.