This paper proposes an experiment made on a portion of decisions of the Italian Audit Court (Corte dei Conti) with the aim to classify the typologies of the amounts included in the text for permitting better searchability by the judges according to the principle "follow the money". The experiment is based on the decisions modelled into the Akoma Ntoso format, one of the most popular XML standards used in LegalXML domain for representing legal knowledge. Akoma Ntoso permits semantics annotation to facilitate Natural Language Processing (NLP) and Machine Learning (ML) approaches fostering legal metadata and document structure. Therefore, it makes possible to promote cross-document analyses with massive amounts of data and correlate events and detect anomalies in diferent legal processes. We introduce a methodology to automate the pattern recognition, information extraction, and document qualification through semantics annotation to improve information retrieval by legal operators within the judicial system and outside. The methodology recognises patterns of interest (PoI) using regular expressions. A supervised ML classifier uses the semantics context of PoI to infer the attribute values of its annotation. The extracted PoI context is used to generate a dataset organized in a tree format in which each level represents an abstraction level of the PoI classification. The annotation is a tagging of the recognised pattern in the text associated with its description as attribute value. The strategy learns the vocabulary of the PoI classification linking diferent levels of a conceptual abstraction. As use-case, PoI consists of monetary amounts associated with their respective origin (attribute), facilitating the identification of common patterns in the rulings of Audit Court, within the same Region/Court and among them.
In the 2022 Rule of Law report [
Digitization is a primordial process to improve the eficiency of data analysis, extracting
information, facilitating their access, and automating their processing through machine learning
(ML) approaches [
This paper contributes to the areas of knowledge mining and information extraction, which
are laborious and error-prone tasks and can be approached through semantic annotations [
This paper investigates the use of Online Learning classifiers for the legal classification of
quantities in the rulings by the Italian Audit Court, on the basis of a legal taxonomy built
on procedural and substantive public law. The methodology used is based on hybrid AI [
We use hybrid AI for merging semantic analysis (e.g., taxonomy, metadata from Akoma Ntoso (AKN)) and ML approach in order to optimize the accuracy, minimise the supervision efort of the dataset by the legal experts, and include elements of explicability for the final legal interpretation. We used a dataset of decisions of the Italian Audit Court in the interval 2000-2015. We have selected about 500 decisions concerning the Emilia-Romagna case-law1
The methodology is the following: 1) all the decisions between 2000-2015 are transformed in Akoma Ntoso for reusing the descriptiveness of the portions of the document (e.g., decision, motivation, facts, temporal metadata, jurisdiction, entities); 2) three legal experts (one senior 2See the Report of the Ministry of Interior of Italy https://www.interno.gov.it/sites/default/files/2022-03/report_ reati_corruttivi_02.2022.pdf; and the European Court of Auditors statistics where Italy has the larger number of staf in Europe devoted to the Audit Court https://op.europa.eu/webpub/eca/book-state-audit/en/#A-9 1The dataset and the code are available at https://gitlab.com/CIRSFID/ai4justice-classifier. The selection has been mainly based on temporal elements to ensure the generalisability of the results in a significant timespan researcher, one young researcher, one lawyer) define a taxonomy of the diferent typologies of the amounts according to administrative law. This taxonomy is based on three axies: (i) the role of the amount in the final decision (e.g., damage or costs for the trial), (ii) the type of harm (e.g., image, revenue), (iii) the source of payment (e.g., salary, pension). The taxonomy is serialized in RDF-Turtle; 3) the paragraphs including the amounts are extracted using NER technique. This step provides inputs for improving the taxonomy with real data (bottom approach). 4) once the taxonomy is accurate enough, it is used by the ML tool for creating a model of classification (62 documents, 239 samples); 5) a new dataset from Emilia-Romagna is used for the evaluation (33 documents, 52 samples dataset); 6) two legal experts (lawyer, young reserch) manually classify this second dataset; 7) the comparison between the ML classification and the manual one produces the evaluation of the indexes; 8) legal analysis of the comparison results is done by legal experts for discovering hidden information for improving the taxonomy or the model.
Pattern mining commonly produces a large set of recurrent patterns that may be classified
according to the semantic context in order to be interpreted properly. Information Extraction
(IE) and Semantic Annotation (SA) often appear together in this type of text-processing problem.
The medical domain [
Semantic annotation is commonly applied to legal document processing but rarely this step is
used as input for ML elaboration. A paper develops a solution to the IE automation focused on
structured knowledge, improving semantic search and ontology generation on textual databases
[
In this section, we describe the main concepts to approach the presented problem – (i) Akoma Ntoso markup language, (ii) the processing of Akoma Ntoso documents, (iii) the taxonomy of the Audit Courts.
The Akoma Ntoso (Architecture for Knowledge-Oriented Management of African Normative
Texts using Open Standards and Ontology) is a XML-standard of LegalDocML Technical
Committee of OASIS (Organization for the Advancement of Structured Information Standards),
promoted by the United Nations Department for Economic and Social Afairs (UNDESA) for
modelling the legal documents. Akoma Ntoso is formed by a set of XML descriptions of
parliamentary, legislative, and judiciary documents, defining legal documents in structured format,
being based on 3-level structure: (i) text, (ii) structure, and (iii) metadata [
Legal documents can be transcribed to Akoma Ntoso manifestations, being structured in (i)
introductory part, (ii) main body content, and (iii) conclusions. In (i), the document presents the
identification data as the document type, number,and issuing authority. Following, in (ii), it is
presented the content lies organized on background of the case (<background>), argumentation
of the judge (<motivation>), decision of the judge (<decision>). At the end, in (iii), the document
is concluded through the closing formulas, dates, and signatures [
Italian Court of Audit decisions are formalised using the type of document of AKN called
’judgement’ that is structured in: (i) <meta>, (ii) <header>, (iii) <judgementBody>, and (iv)
<conclusions> [
The taxonomy of the typologies of amounts of the Italian Audit Court has the aim to describe the role that each amount plays in the final decision of the judge according to public law. We used four criteria: a) the role played in the procedural law of the "Code of Accounting Justice"; b) the type of injury; c) the provenance of the sum; d) the accumulation of the amounts. The heterogeneous typologies are in this taxonomy simplified with the goal to ofer to the judge a new instrument of analysis of the case-law and also to perform, in the future, network analysis of the relationships between diferent case-law. The first used criterion is based on the teleological role of the amounts involved in the litigation: expense (Spesa) or State treasury sum (SommaErariale). All the nomenclature used (in Italian) is based on the Code of Accounting Justice3. The “State Treasury Sum” (SommaErariale) top-class represents the typologies of each income or payment in the treasury cofers, including the values that are only hypothetical because they were determined as a precautionary measure. The “Expense” (Spesa) top-class represents the disbursement approved by the judge and charged to all parties or, more frequently, to the losing party. The subclasses "Defense", which means defense costs, (Difesa), and "CourtFee" (Giudizio), which means the costs for the ofices, specify the nature of the disbursement by indicating the service that justifies it. Taking into account the competence for matters attributed to the Audit Court by art. 103 of the Constitution and of the various phases of the accounting process, it possible to give the sub-classes name (Compensation, Credit, Indebtedness, Indemnity, Sanction, Withholding, Sequestered Value, JudicialAccount) representing the sum according to the object of the dispute. The sum, for example, is classified 3See also http://www.normattiva.it/eli/stato/DECRETO_LEGISLATIVO/2016/08/26/174/CONSOLIDATED as a "Credit" when it expresses the economic value of a credit right which is the object of the dispute in the pension proceedings. The typology based on the source of the amount is classified here using “Salary” (Salario), “Pension” (Pensione), “MovableProperty” (BeneMobile), “ImmovableProperty” (BeneImmobile). It ofers a second criterion for a further specification of the state treasury sum, which, in addition to acquiring characteristics from the transit in the process, afects the components of such movable and immovable property in the case of the sanction and the salary and pension in the case of withholding. This relationship is managed using isAppliedTo. Another important class is “Compensation” (Risarcimento) which introduces the third criterion of analysis: the typology of injury. We have "Damage" (Danno) class, which is connected with the top-class with the relationship isRestoredBy. It expresses the economic value of the injury sufered by the injured party, which the sum qualified as "Compensation" intends to restore. The "Equity" and "Non-Equity" sub-classes, in compliance with the provisions of the Italian Civil Code, characterize the damage on depending on the type of interests harmed by the damage (e.g., property rights, personality rights, etc.). In the light of the type of interests harmed by the damage, the jurisprudence then identified the damage from disservice and damage to the image corresponding to the subclasses "Disservice" and "Image". It must be emphasized that the latter classes, of jurisprudential derivation, are more subject to modifications and interpretation. The boolean hasMultipleEnumeration expresses the fourth criterion: the possibility that the compensation is single or cumulative or that the amount is representative of a single compensation item or is the sum of several compensation items. By way of example, the provision could indicate the amount of 100 as total compensation in which 30 is justified by the compensation for the damage to the image and 70 for the damage from the disservice.
In AKN context, tagging patterns evolve two steps: (1) to extract patterns of interest (PoI) from textual data using regular expressions, and (2) identify the values of the PoI attributes. In (2), attributes are identified through data mining using semantic contexts of PoI as input to ML approaches. Figure 1 shows the possible values of the PoI attributes organized into a taxonomic tree.
In this work, PoI identifies monetary quantities (<quantity>) cited in a legal sentence, being defined as {"?:valute": ["(?:£||Euro|euro|EURO) "?:value" : ["\\d+\\.?"], "quantity": ["{{valute}}(?:\\s*)?{{value}}
+((?:\\,\\d+))?"], "patterns": ["{{quantity}}"]} (?:\\.)?" ], The <quantity> tag is expressed by a regular expression using JavaScript Object Notation (JSON). The monetary values can be found in euro (€, ‘euro’, ‘EURO’, or ‘Euro’) or Italian lira (£). It can be followed or not by a dot (‘.’), and by an undefined sequence of spaces ( ? : ∖∖* ) before achieving the numerical value. In sequence, there is at least one numeric digit (∖∖+), followed or not by a comma (‘,’), and at least another numerical digit (? : ∖∖, ∖∖+). The ? : operator is a group constructor, and a symbol followed by an interrogation mark (?) informs the possibility of reading the symbol, but not the obligation. Some <quantity> examples are €20.561, 83, £.82.967, 92, and euro 888, 10. In the tagged text, e.g. €215, 10 will appear as <quantity class="Difesa"> 215,10</quantity>, in which ‘class’ is the attribute of <quantity> that identify the money origin. The attribute ‘class’ can be associated with any of the possible classes.
The taxonomic tree (TT) provides an abstraction control mechanism to classify PoI using its semantic context. Figure 1-a shows the <quantity> taxonomy in a graph, in which the nodes (legal concepts) are linked with diferent relationships. To make possible intermediary classification systems, the taxonomic graph was mapped to a taxonomic tree (Fig. 2-b) in which the parent-child nodes are linked using class-subclass relationship. With this modelling, an element of a child node is also an element of its parent, except for when the relationship is not diferent. Suppose a set of concepts organised using the abstract data type tree to define the PoI taxonomy, in which the closer a concept is to the root, the more abstract its definition, and the closer to the leaves, the more concrete its meaning. In this sense, the root is the most generalized TT class, and a leaf is the most specific one of its branches. Each TT concept defines a possible class of the PoI taxonomy. A data sample classified as belonging to the root class means that the sample is poorer in semantic context information than a sample classified as belonging to a leaf class. A tree branch is defined by the way between the root and a leaf, in which the abstraction level of the classes decreases on the way. Data samples are mapped into the set of possible classes = {1, 2, . . . , , . . . , }, in which is a TT node (and a legal concept) and = 29 in this work. As we can see, there are two special kinds of nodes: the root, and the leaves. The root, the unique node characterised by has no parent; the leaves, the nodes that have no children. TT is composed by a set of legal concepts associated with the <quantity> PoI. Each tree level represents an abstraction level in which a sample can be classified, having a set of possible classes , in which = 0, . . . , , and is the maximum level of the tree ( in this work, = 5). is a subset of and defines the possible classes of a classification system associated to the tree level . For example, 2, the second level classification system, has the {‘JudiciaryAct’, ‘StateTreasurySum’, StateTreasurySumNonLitigation, ‘StateTreasurySumLitigation’, ‘Expense’, ‘DefenseFee’, ‘CourtFee’} possible classes, as we can see in Fig. 1. For this example, each sample classified using the 2 classification system will be associated with one class of 2. In this work, we have 6 classification systems, {0, 1, 2, 3, 4, 5} with 1, 3, 7, 15, 27, and 29 possible classes respectively. Each legal concept can be seen as a discrete sequence of semantic abstraction classes, from it until the root, e.g. a data sample classified as ‘Image’ is also classified as ‘NonEquity’, ‘Compensation-Damage’, ‘StateTreasurySumLitigation’, ‘StateTreasurySum’, and ‘JudiciaryAct’ since the ‘Image’ class is contained in the ‘NonEquity’ class that is contained in the ‘Compensation-Damage’ class, and so on. The multi-classification system is modelled to be a legal support tool. A <quantity> sample is classified in the most specific-concept classification system (5), orbiting in all the legal concepts defined by its branch, the path formed between the root and its most specific classification, that can be a leaf but not necessarily. The hierarchical concepts provide an abstraction scheme that will be used by a decision-making support tool for judges and legal analysts, helping them to analyse the <quantity> origins, enabling them to identify types of money flows. In this sense, if someone needs to understand a legal process in a more general view, the abstraction level of the information can be increased using a more general schema moving towards 0, and if it is necessary to be more specific, the abstraction level can be decreased, floating towards 5.
Context-driven Neural Networks is a fully-supervised one-layer Neural Network (NN)
constructed incrementally from an online data stream [
In Fig. 4, the neuron receives x and compares each with the terms of the dictionary until it finds a coincidence. If is in , is used in the aggregation function . The comparison between and the terms is crisp, therefore the result can be "0" if there is no match, and "1" if is in . The comparison output is weighted by the frequency value in , generating the aggregated output.
The output is the expression of the membership level of x in . is an aggregation function that choose the best candidate, * , to be the x class, ^ * . is ([1, . . . , ]), in which is the operator.
The aggregation functions are the same for a NN setting with the exception of that is the S-norm in all configurations.
The naive version of metric is presented [
The ranks the item based on its importance , on the document , i.e., the importance of in , = , (1) being , the number of occurrences of the term in the document ; and the total number of items in . The counts the number of relevant documents in the collection related to the term ,
, = , = ︂[
| ∈ : ∈ | | ∈ : ∈ | ︂] , ︂] + 1 , 1 + | ∈ : ∈ |
︂] + 1 , and , =
︂[ being referenced as tfidf2, tfidf3, and tfidf4 in Fig. 7- 6. Each word in has an associated TFIDF value that is used as weight in , in which is the average value of of in . , =
1 + | ∈ : ∈ |
︂]
in which is the collection of documents with | | = , the number of known classes, and
1 + | ∈ : ∈ | indicates the number of documents in which the term appears at least
once. is the result of the multiplication of the terms. The higher the , and the
,, the higher the importance of the term in , i.e., the term must be frequent in a
small number of documents in to be ranked as important [
In this work, it is used 3 variations of the presented formula ︂[ (6) (7) 6.1.2. P-Norm P-norm is defined as = P-Norm. 6.1.3. S-norm maximum operator, ||w|| = ⎝ ⎛ ∑︁ | |⎠ =0 ⎞ 1 in which = {1, 2, 3, 4} in this work, being referenced as 1-norm, 2-norm, 3-norm, and 4-norm in Fig. 7- 6. Each word in has an associated frequency as , that is used as weight in S-norms () are commutative, associative and monotone operators. The used S-norm is the (w) = =0,..., ,
max that is used as weight in = S-norm. being referenced as max in Fig. 7- 6. Each word in has an associated frequency as ,
In lines 1-3, the accuracy vector is initialized, the level tree and the aggregation function are read, and the neural network is created. In lines 4-13, there is the [ℎ] stream classification: in the line 9, receives [ℎ] and returns the estimated class ^* . The is updated with the new information (x, )[ℎ] in the line 10. The online accuracy is calculated in line 11 according to Eq. (8).
Online Learning: Context-Driven NN Classifier
This section describes the data gathering process, the annotation, the datasets and the metrics used to report results.
The dataset is composed of information extracted from legal rulings of the Italian Court of Audit from 2000-2015 of the Emilia-Romagna region with 584 samples in total, 327 judgments. Rulings are presented as Akoma Ntoso-formatted documents, from which are extracted monetary values from the textual data of their ‘Decision’ parts. The dataset is organized into the columns <filename>, <quantity>, <Decision_Part>, <Semantic_Context>, and <Class_Expert>, in which: • <filename>: the name of the file of the legal sentence, • <quantity>: the extracted monetary quantity, • <Decision_Part>: the text from which the <quantity> sample was extracted, • <Semantic_Context>: a list of words, a maximum of 20 words apart from <quantity>, extracted from the <Decision_Part> text, characterizing semantically <quantity>, and • <Class_Expert>: one of the classes described in the taxonomy that is associated with <quantity>.
Considering the (x, )[ℎ] stream, <Semantic_Context> is x and <Class_Expert> is .
One legal expert annotated the 584 samples by identifying the correct class at the lowest possible
level of abstraction, i.e., by preferring a more specific class over a broader one, when possible.
Such possibility is ofered solely by the context, as detailed by the decision. Following the
methodology used in other works, e.g., [
The data distribution is shown in Fig. 5-a. The y-axes shows the class name and the x-axes, the number of samples. The partial data distribution (PDD) considers the amount of samples of each class and, on the other hand, the total data distribution (TDD) takes into account PDD added to the recursive sum of all its children samples. Figure 5-b shows the distribution of the quantity of samples through the set of classes, informing how many classes have the amount of samples into a given range. For example, it is considered ranges with widths of 10 samples, i.e., from 0 to 10, from 11 to 20, and so on. As it is possible to see, the problem is an unbalancing classification in which 18 from 29 classes have until 10 samples. In Fig. 5-c, it is shown how the size of the semantic-context list is distributed through the dataset, approaching roughly a Gaussian distribution.
The data distribution makes it dificult to classify data with high accuracy because it is available a highly unbalanced small data. To improve the classification quality, it is proposed a multi-classification scheme in which each tree level represents a possible classification problem. Based on the fact that an element of a subclass is also an element of its super class, the classification system is composed by 6 classification problems with 1, 3, 7, 15, 27 and 29-classes, in which it is available 584 samples distributed over the possible classes in each one of the problems.
Online classification accuracy, ∈ [
We evaluated the Context-driven NN considering 584 <quantity> samples. We considered 9 diferent aggregation functions, 2 algorithm versions, and 6 classification systems, i.e., 108 combinations of hyperparameters. Each combination was repeated 20 times, being executed 2160 simulations in total. Table 1 shows the values of the hyperparameters used in the simulations. In the aggregation functions, , tfidf and p-norm have 4 variations each one.
As a side efect of the Pigeonhole principle, increasing the number of possible classes, decreases the method accuracy as shown in Fig. 6-(a) and Fig. 7-(a,b). The probability () to choose the correct class by chance is 1 , in which is the number of possible classes, and for the 1, 3, 7, 15, 27, and 29-classes problems, the probability will be 1, 0.33, 0.14, 0.06, 0.037, and 0.034 respectively, and their () rate is in the order of 1, 2.51, 5.57, 10.67, 14.95, and 16.51. As we can see, when increasing the number of possible classes, the impact of chance on the classification becomes smaller. Since the taxonomy is structured as classes and sub-classes, the probability of misclassification among the classes of the same branch is bigger than between diferent branches. As we have data extracted from free-style text, the juridical terms used to describe a <quantity> sample can vary greatly, contaminating x with words with no semantic importance to identify the correct class or with words that are useful to identify other possible classes. Table 2 abstracts the results of the simulations considering our best performance using tfidf3 with 99.5% of confidence. The accuracy of the ofline algorithm version (94.18%, 61.53%) is slightly higher than the online one (89.87%, 57.02%). It is easy to understand the results if we see Fig. 6-b which describes the behaviour of the accuracy on-the-fly for the online NN version. In the beginning of the learning process, the accuracy is unstable, tending to converge to the method accuracy over time. The ofline NN has a training phase which includes the beginning of the learning process. In this way, it is expected that the testing phase has better accuracy than the training phase in this case.
If we compare Fig. 6-a and Fig. 7-(a,b), it is possible to see that the confidence interval, expressed by the grey area around the lines, is slightly narrower in Fig. 6-a than in Fig. 7-(a,b) since the online version uses the samples used by the training and testing phases together. Figure 6-b shows the online accuracy of 3-classes and 29-classes problems, providing the asymptotic limits to the accuracy of all classification systems. Figures 6-7 present a similar behaviour of the accuracy of the classification systems, being unquestionable the superiority of the results related to the TFIDF aggregation functions family, the unique approach exclusively NLP.
Following the methodology, we analysed the most frequent errors from a legal point to check whether the taxonomy was tailored to the text under scrutiny or whether adaptations were needed. The analysis was carried out on each experiment with similar results. For reasons of brevity, the following detailed assessment only concerns the experiment with the lowest accuracy. Fig. 8 shows the 7 most frequent errors for the 29-class experiment and their relative number of misclassification, which account for more than 50% of the total number of errors. In the most frequent error, the algorithm classifies "Expense" instead of "CourtFee". Therefore, it is unable to go from a parent class to a child class. The reasons underlying this error may be linked to the absence of clear linguistic formulae that identify the costs in Court in their general meaning with respect to the costs referred to in the decision. Reading the misclassified cases, two problems are detected: 1) quantities and judicial expenses are mentioned consecutively in the decision, thus the linguistic formulae influence the classification negatively; 2) the formula "spese compensate" ("compensated expenses”) significantly afects the classifier. The taxonomy could be refined - in the future - by adding a boolean specification on the compensated expenses. With regards to the second and third most common errors, it can be observed that the classifiers chose a "child" ("Equity") node rather than a "parent". One possible explanation of this recurrent mistake is that the classifiers associate some linguistic patterns to the "child" class that are hidden from the annotator, who opted for a more abstract term. However, it has to be noted that misclassifications occur on subclasses that have some degree of connection to the right class. The same kind of misclassification can be observed for the fourth, fifth and seventh most frequent cases, but these are more likely to be justified by the lack of specific data. The sixth case is similar to the first one, and can be explained by the presence of the word "pensionistico" ("about retirement/pension") in the sentence, which might trigger the wrong case. However, the context did not allow the annotator to define the quantity as related to the pensions.
In this study, Online Context-Driven Neural Networks were tested in classifying quantities from decisions of the Audit Court, also in comparison with ofline algorithms, in combination with TF-IDF aggregation functions. From the legal evaluation of the experiment, it emerges that the class "CourtFee" is particularly problematic due to its frequency (5th most represented class overall, 1st as initial class) and the lack of a clearly-identifiable linguistic pattern to be associated with this class. This is a rather expected result given the correlation between these two factors. In further studies, a refinement of the ontology could identify some implicit instances of this class. Similarly, expanding the taxonomy with regard to other forms of judicial expenses, including the compensated ones, could be helpful in increasing the mapping of the quantities to be labelled. Finally, a cross-validation that includes other regions and other courts could be a viable and valuable avenue for further research. In any case the proposed technique is quite promising for classifying the amount using a taxonomy and so to improve the categorization of the diferent decisions also on the basis of the role plaid by the quantities in relation of the type of damage. In particular it is possible to make complex query like "give me all the decisions where the quantity of the damage is > X, concerning ’Disservice’ AND where the CourtFee is >Y". Additionally the Online Context-Driven Neural Networks permits to adequate the ML model dynamically according with the evolving of the language of the decisions in relation of the jurisdiction (e.g., diferent regional language) or due to the historical period (e.g., diferent terminology) and the legislative changes.
This work was supported by the ER4JUSTICE - NEXT GENERATION initiative (CUP E44I20000270002) for the eficiency of the judicial system, funded by Regione Emilia-Romagna and coordinated by Fondazione CRUI. Press, 2019, pp. 205–210. [26] L. Decker, D. Leite, D. Bonacorsi, Explainable log parsing and online interval granular classification from streams of words, in: 2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, Padua, Italy, 2022, pp. 1–8. doi:10.1109/FUZZ-IEEE55066. 2022.9882710.