Automatic analysis of legal texts is increasingly of interest to address the issue of interpretation and compliance concerns. This paper proposes a two-step framework to investigate implicit relationships in legal documents, starting with a corpus-based approach. By introducing an annotation process, the goal is to obtain a gold standard corpus suitable for machine learning experiments. In a second step, we propose a set of features to perform the task of predicting relationships between parts of a norm, as a way to improve legal interpretation. We discuss our rst results concerning the annotation task, as well as the adoption of graph-based measures derived from social network analysis. We perform a practical application to an European Union regulation. The proposed framework exploiting network analysis in addition to a corpus-based approach can be applied to address a binary classi cation task.
Computer technologies and systems are growing rapidly in di erent elds,
including the legal domain [
This paper addresses the speci c topic of legal interpretation, de ned as the fundamental activity giving meaning to legal documents. The goal of interpretation is to determine the normative messages that arise from a speci c legal text. We investigate the subject by focusing on the relationship between internal parts of legislation. In fact, the identi cation of existing links is at the core of the interpretation process. Moreover, the automatic extraction of information is a challenging task in order to analyse inter-relationships between norm parts. Our work proposes a methodological framework to address an automated system able to consider both the explicit and implicit information in legal text. Therefore, a primary e ort necessarily addresses the understanding of the meanings conveyed by the language. In our experience, this is a typical problem that exploits annotated corpora and related tools for the analysis. Furthermore, we propose the investigation of network analysis by shaping inter-relationships as a graph.
In particular, our proposed framework follows a two-step approach to address
the issue. Firstly, a corpus-based approach identi es di erent types of
interrelationships in a norm. This initial step includes the de nition of guidelines
to address the human annotation e ort. The output of this process includes
the identi cation of classes or labels, in order to obtain a gold standard
corpus [
In a second step, the exploration of a graph-based analysis can help both to
better understand the inter-relationships occurring in the norm and to improve
the feature set that can be used in a classi cation task. Moreover, several metrics
from social network analysis are compared to the annotation output. In this
respect, our research problem is: in a general NLP framework to investigate legal
text, are graph metrics helpful to investigate inter-relationships in norms? This
work resumes an ongoing experience in developing NLP resources and tools for
legal text [
The paper rst reviews background and related work in Section 2 followed by a description of our corpus-based and graph-based approach in Section 3. We describe the initial results of the proposed system to tackle the research problem in Section 4. We conclude the paper with some future work. 2
Legal interpretation is a well-known mechanism from law for adapting norms
to unforeseen situation [
Similar work focused on a network perspective concerning systems of
interrelated authoritative legal texts. The detection, resolution and labeling of citations
in the legal domain was addressed in [
This paper explores European legislation. Directives, and regulations to a
lesser degree, are prescriptive but su ciently general to allow addresses (usually
member states) to articulate their own detailed norms and procedures as they
prefer in order to achieve the goal(s) of the legislation. Not only is this kind of
legal text typically goal-oriented, it is also particularly given to principle-based
(balance) rather than defeasible reasoning. The initial part of the document
consists of recitals, which are ostensibly explanatory and do not have the same
status as normative provisions. The relationship between recitals and normative
provisions is a debated topic relying on di erent doctrinal positions [
Our framework combines a quite traditional approach starting from the creation of a gold standard corpus in the development of machine learning experiments based on Natural Language Processing (NLP). This is a prerequisite to perform a classi cation of relationships in legal texts. In our initial e ort, we opted for manual annotation to produce a corpus labeled with classes of interest. Annotation is an important task for NLP, where the common pipeline includes the following steps: identi cation of classes or labels; the de nition of detailed guidelines; training of human annotators; manual annotation of the corpus; and nally computation of inter-annotator agreement.
Identi cation of labels The annotation of relations in legal text includes a scheme's de nition and its application to the legal document. The aim of the scheme is to clearly de ne the kind of information which must be annotated. This phase includes also the inventory of markers to be used, as well as the annotations granularity. This step relies on the e ort of experts in the speci c domain of the considered documents.
Annotation phase At this stage, it is very important that each annotator
works independently. In this way there will be evidence of any situations of
disagreement, which will be resolved later. In a rst step, the annotation can be
presented in a spreadsheet in order to facilitate the work of annotators. For
instance, legal text can be split into di erent structural parts such as articles or
paragraphs for comparison; and di erent Excel sheets can be used to
meaningfully separate di erent areas of comparison. It can be useful to ask annotators
to keep track of their start and end time in order to calculate the average time
required for the particular annotation task, Inter-annotation agreement can be
measured using metrics such as Cohen's kappa [
Card sorting Commonly used by information architects, card sorting is a
popular user-centred method aimed at the identi cation of patterns among data
[
Natural Language Processing By following a quite traditional NLP pipeline, the aim of this part is to start with preprocessing the legal steps to obtain stems of terms. In linguistics, stemming means reducing words into their corresponding root form. For instance, it will be possible to compare singular and plural forms of a same term occurring in di erent parts of the legal text. Typical passages in preprocessing include conversion to lowercase characters, and removal of punctuation marks and stop-words. The next step is tokenisation, in order to separate terms into tokens, followed by stemming. Further analysis is possible by detecting parts of speech (POS) such as verbs, nouns, adverbs, adjectives and so on. The text processing phase can be performed by using common programming languages (e.g., Python or R) with well established NLP libraries.
Identifying features The aim of the second step of the framework is to automatically extract a set of features to be used in machine learning experiments. In particular, by following a Bag-of-Words model, several features can be considered to represent the text. In addition to traditional N-gram models (sequences of N words), several other items can be extracted from text, e.g. the frequency of di erent parts of speech, the sentiment polarity of di erent words (e.g. whether they belong to `positive or `negative dictionaries), word and character length, term frequency (using TF-IDF).
Network analysis We perform a Social Network Analysis on two kinds of
graphs. A rst graph connects norm types depending on the co-occurrence of
terms. If the stem of a term is detected in di erent parts (e.g., Recital 2 and
Article 6) of a legal text, then an edge will connect the two parts. The weight
of the edge represents the total amount of co-occurrences. This kind of graph
will describe inter-relationships between norms from a linguistic perspective. A
second kind of graph explores the role of terms in the document, by linking
different stems if they co-occur in the same norm type. The number of times they
both appear in the same part of the document will be the weight of the edge.
E.g., an edge between Stem X and Stem Y weighted 3 indicates that Stem X
and Stem Y co-occur in the same norm type three times. Social network
metrics from this graph can be adopted as individual features for the classi cation
e ort, e.g. degree, betweenness centrality [
Weighted multi-layer networks A further investigation of the topic can
benet from the analysis of a multidimentional or multi-layer network. These kinds
of network have been receiving increasing attention in the last few years. The
opportunity to shape inter-relationships in legal text can be related to the
classi cation e ort. As there exist di erent labels for di erent kinds of relationship,
each label can be a dimension to distinguish a network. In this respect, a
legal document may include di erent networks containing multiple connections
between any pair of vertices. Then, each label denotes a di erent layer, both
in the case of a network of stems and norm type. Recent studies introduced
a framework to investigate such complex networks having an additional degree
of complexity provided by multidimensionality. For instance, a recent work [
Classi cation The contribution of this phase is to formulate an experimental setting in terms of a classi cation task. The existence of a relation between two parts of the document can be explored with a supervised machine learning experiment carried out by using a model trained with the annotation results. In particular, the framework includes several binary classi cation tasks, where the classes (or labels) are: i) the existence of a relation of any kind; ii) the existence of a relation of a particular kind (one for each label identi ed at the beginning of the corpus-based approach. Several classi cation algorithms can be adopted, e.g. nave Bayes, logistic regression, decision tree, support vector machines. For instance, in this phase the gold standard corpus can be used to train a binary support vector machines with the labelled relationships.
Evaluation Performance of the classi er can be evaluated by computing the Fmeasure, which provides information on accuracy based on the ratio between precision and recall. Cross-validation can be applied to evaluation. For instance, in ten-fold cross validation the process involves repeating ten times the following steps: i) break the training data into 10 equally-sized partitions; ii) apply the learning algorithm on nine parts, while testing on the remaining folds. The nal measure is the performance average of the ten parts. Once an estimation of the model performance has been obtained, this can be applied to new data. 4 4.1
First of all, two legal experts identi ed eight di erent kinds of relations between
recitals and (sub-)articles based on their analysis of Directive 2004/23/EC in
the Italian language. These are: Conceptually Similar (whether using the same
or di erent wording), Constitutive (linking norms containing de nitions of legal
terms to norms containing those terms), Motivation (where one norm provides
the principle or goal that motivates another norm), Impact (in terms of con
icting goals that may restrict one or both of the norms, or norms for enforcement
or monitoring that impact on the e cacy of classic deontic norms), Indirect
Internal (norms A and C are linked indirectly where norm A cites another internal
norm B which is related in another way to norm C), Via Other Law (a norm that
is related to another norm which cites another law and cannot be understood
without reference to that law), Procedural (linking a norm describing a
procedure by an EU institution to support the goal of another norm), and Contextual
(linking deontic or other norms to norms that provide contextual information
such as jurisdiction and entry into force), and Norm Group (where two or more
norms are connected due to being part of the same general requirement). For
space reasons, we refer the interested reader to the full descriptions and
examples provided in [
An initial result of the corpus-based approach is the creation of a document including the annotation scheme provided by an expert of the domain. This document o ers a clear de nition for each case as well as a corresponding example. These indications become the shared guidelines for independent annotators. 4.2
Two annotators were involved in this initial e ort. Each annotator was provided with a spreadsheet le. At the top of each sheet there are spaces to indicate the start and end times concerning the annotation activity, as well as the number of comparisons made. Each annotator was asked to indicate the time spent on multiple work \sessions".
The document (in Italian) is accessible at the following link: www.di.unito.it/ ~sulis/NLPxLAW/LineeGuida_ClassificazioneTipiLegami.pdf To investigate the annotation phase, we looked at inter-annotation agreement on the relationship between three recitals from the considered law, and their relationships with all the articles, or sub-articles where present. Those three recitals were the rst, the shortest and the longest ones. Annotators had to decide which kind of relationship exists (if any) between, for instance, the rst Recital (henceforth R1) and Article 1, paragraph 1 (henceforth A1.1), and the type of relationship. If there is a relationship of any kind, a value of 1 is provided, while a value of 0 means no relationship exists. The results can be expressed as a sequence of comma separated values, e.g., a list of triplets for each type:fR1,A1.1,0; R1,A1.2,1; R1,A1.3,0; etc.g.
For analysis of the results, we shall rst look at inter-annotator agreement on whether a link exists at all between the recitals and (sub-)articles studied (see Table 1). There are: 86 cases where both annotators agree on whether a link exists, 54 cases where the annotators agree on the existence of a relationship; 32 cases where both annotators agree on the absence of a relationship; 28 cases of disagreement - 22 cases where only the rst annotator considers that a relationship exists; and 6 cases where only the second annotator considers that a relationship exists. This provides a percentage of agreement of 75.4%, and a Cohens kappa agreement of 0.5. However, this encouraging result is balanced with signi cant di erences in the labelling of individual kinds of relationships. In fact, there are very di erent values in the distribution among classes (Table 2).
While some classes have been identi ed with a reasonably similar number of times, others are very distant in their allocation. Qualitative analysis of the results imply that some problems arise due to di erences in the directive on which the classi cation scheme was modelled and the regulation that was annotated. For example, Table 2 shows signi cant disagreement on the allocation of the Impact label. One annotator extended the Impact category to also include the impact of planned future guidelines, in consultation with member states and other parties, on the interpretation and e cacy of the stated goals of the regulation. The more legislation are annotated, the greater will be our understanding and ne-tuning of the classi cation scheme. A more expected source of di culty was the Via Other Law link: determining whether it should be necessary to consult the referred legislation to fully understand the article or recital in question. A nal point that is clear from the data is that the most common type of link was Motivation. This is in common with the accepted purpose of recitals providing explanatory notes to aid understanding of the substantive provisions.
The output of this phase clearly demonstrates the di culty of the task, as well as the need to re ne the labels and revise the annotation guidance. We also conclude that the provision of annotation guidelines is not su cient to train annotators in this novel classi cation scheme. For further annotation exercises, we intend to conduct seminars featuring both typical and hard cases in order to ensure proper understanding of the annotation task. 4.3
Our annotation scheme includes recording the start and end time of annotation sessions by each annotator. The average execution time for each annotation session was an average of 62 minutes. Considering the above-mentioned three Recitals, one annotator was faster then the other but split his e ort in multiple work sessions: 6 sessions with an average time of 24 minutes, instead of 3 sessions with an average time of 99 minutes. 4.4
The card sorting exercise was carried out by the same people who acted as
annotators. Even if this task has not yet been completed, some interesting insights
have emerged from the material preparation phase, highlighting the need for
methodological adjustments in comparison with traditional card sorting. Firstly,
legal experts who acted as our consultants for this activity have pointed out that
single parts of legal text can in principle be part of several norm groups. Thus,
we had to prepare multiple cards for each recital/paragraph considered.
Secondly, we observed that the number of di erent cards we obtained was higher
than the number of cards commonly used in card sorting, and the content in
each card was longer as well. Therefore, we plan to assess the annotators'
cognitive load, by using both quantitative measures such as NASA-TLX [
For processing legal text, we applied typical techniques to extract useful
information. For instance, by performing POS tagging on our EU regulation, we
computed 1,636 words, with the prevalence of Nouns (1,239 occurrences),
Adjectives (171) and Adverbs (58). In addition to typical NLP analysis, we reduced
our document (recitals and (sub-)articles) by transforming words into their
corresponding stems. We adopted the Python NLTK library [
Graph of Recitals and Articles A rst graph investigates the relationship between recitals and articles considering each of them as a vertex. This kind of representation is the one on the left in Figure 1, by including only the vertices with a degree higher than 10 to improve the readability of the network. Edges are proportional to their weight, i.e. the number of terms co-occurring between the two corresponding vertices. These values may represent the strength of the relationship between the two parts of the law. Graph metrics indicates here the strength of the relation. Once the annotation e ort of implicit links has been completed, it would be interesting to correlate the results about interrelationships of a legal text with the corresponding graph metrics, i.e. degree or centrality measures. We hypothesize a certain similarity between the lexical graph-based measures and the existence of a some kinds of implicit relationships (e.g. Conceptually Similar, Motivation) between di erent parts of the text.
Graph of Terms A second graph analysis explores the relationships among terms, after the stemming process. By considering stems as vertices, the edges represent the co-occurrences in the legal text under consideration. Figure 1 (b) describes a sample of this kind of graph by considering only vertices having a degree higher than 100. The weight of an edge represents the co-occurrences of stems in the document. Several metrics are computed for each vertex to measure the corresponding relevance in the graph. With the aim of assessing the relevance of the vertex in the graph we report two measures: degree and centrality. Degree varies in a range from 1 to 68, with an average value of 52. Stems with higher betweenness centrality refer to both the topic of the current law (i.e., \Medicinal", 21.8; \Pazient", 16.9) and common terms (i.e., \Tal", 21.9; \Scop", 17.1). The representation of each part of the legal text can be addressed with a bagof-words approach, or including n-grams to capture more context around each term, as well as POS-based features (e.g., the number of nouns, adjectives, etc. in each part). The output of this step consists in a set of features collected as a numeric feature vector. The goal is to obtain a feature model to perform binary classi cation. We add to these traditional approaches the information derived from graph analysis. By considering the graph with stems as vertices, each vertex metric can be used to create a feature of the corresponding parts of the legal text (e.g., articles or recitals). For instance, the average value of vertices' metrics (e.g. degree, betwenness centrality, closeness centrality) can be exploited in the classi cation step. Similar considerations concern the graph of relations between stems co-occurring in di erent parts of the legal text. In this case, the nal set of features may include network metrics depending on the network topology. 5
This work proposed a framework to develop an automated system to nd
interrelated parts of a norm. These links may help legal professionals to interpret legal
issues for speci c cases. After describing the framework, we described a practical
work-in-progress application carried out at the University of Turin (Computer
Science Department and Law Department). In our data-centric framework we
address the task of automatically identifying implicit internal links between norms
in the same legislation. In particular, we introduce here the development of a
corpus of annotated links mapping recitals and articles. We initially applied our
framework on EU legislation. This kind of legal text is deliberately imprecise,
to allow members to ful ll the objectives in their own way. The rst analysis of
this step carried out by two annotators clearly shows the complexity of the task:
while the agreement seems quite satisfactory in the identi cation of inter-related
parts, the disagreement on the types of relationship is very high. In addition to
the corpus-based approach, we took the opportunity to operate a classi cation
task. The experiment can exploit a set of features extracted from traditional
NLP approaches. In addition, we explored here the adoption of features from
a graph analysis to improve the results. In this framework, graph-based
metrics concerning vertices can be of interest as a feature of a machine learning
experiment. As a future work, we plan to adopt such metrics to perform binary
classi cation tasks. The classes will be those adopted in the annotation phase,
i.e. the presence or absence of a link as well as the kind of existing
relationship. Special attention will be given to multidimensional network analysis [