In this paper, we present the ASKE (Automated System for Knowledge Extraction) approach to legal knowledge extraction, based on a combination of context-aware embedding models and zero-shot learning techniques into a three-phase extraction cycle, which is executed a number of times to progressively extract concepts representative of the diferent meanings of terminology used in legal documents chunks. We show ASKE in action in a case study of legal knowledge extraction from a real corpus of case law decisions in the framework of the NGUPP project.
To cope with the growing volume, complexity, and articulation of legal documents as well as to
foster digital justice and digital law, increasing efort is being devoted to AI-based techniques for
legal knowledge extraction. The availability of techniques for extracting knowledge from legal
documents is not only desirable but even necessary, and the benefits and concrete outcomes
that could result from the difusion of such technology are many and diferent for both legal
practitioners (i.e., lawyers, judges and Courts), administrations, and general public. Legal search
through legal knowledge extraction is an extremely important instrument for legal practitioners
in both common law [
In this paper, we present ASKE (Automated System for Knowledge Extraction), an approach to legal knowledge extraction with focus on abstract concept discovery using a combination of context-aware embedding models and zero-shot learning techniques. ASKE takes a corpus of legal documents as input and it extracts a graph of concepts which are used to classify the given documents at the chunk-level (e.g., paragraph) granularity. Through context-aware embedding, document chunks and concept definitions are projected in the same semantic space, to appropriately capture and manage the meaning of legal terminology by taking into account the context in which terms are used. Through zero-shot learning, a multi-label classification process is performed in an unsupervised way, without relying on any pre-existing annotation of legal documents. The distinguishing feature of ASKE is the implementation of a cyclic extraction process that, at each cycle, progressively incorporates newly extracted legal knowledge into the ASKE Conceptual Graph - ACG, a graph-based data structure initially populated through a data preparation step.
After describing the ASKE process in a nutshell, we show ASKE in action in the context of two use cases of knowledge extraction considering a corpus of 50 Italian case law decisions in the framework of the Next Generation UPP (NGUPP) project. NGUPP, funded by the Italian Ministry of Justice, aims at providing artificial intelligence and advanced information management techniques for the digital transformation of Italian legal processes and digital justice in general.
ASKE is conceived to build a conceptual view over a considered corpus of legal documents. A data preparation step is initially executed, and it is followed by an iterative three-step extraction process characterized by i) document chunk classification , ii) terminological enrichment, and iii) concept derivation (see Figure 1).
Data preparation consists in the application of conventional text processing techniques,
that are tokenization, lemmatization, and embedding. Tokenization has the goal to separate
a document into chunks. A chunk represents the text unit to consider for classification
and it determines the granularity of the document that can be associated with a concept. A
chunk consists of a few sentence/phrase detected in a document, up to a maximum size of 512
words1. After tokenization, the terms appearing in chunks are lemmatized and a vector-based
representation of each chunk is finally built. To this end, a chunk is associated with a set of
terms therein contained. Any term ∈ is described as = (, , ¯) , where is
the label of the term (i.e., the lemma), is a description of the term taken from a reference
dictionary/vocabulary (e.g., WordNet), and ¯ is the corresponding vector-based representation
1The size of the document chunk is experimentally determined according to the features of the considered corpus. A
chunk should be large enough, so that the context can be captured, but not too much extended to avoid segments
that are long to read and potentially noisy due to the presence of multiple concepts.
according to Sentence-BERT [
For triggering the knowledge extraction process, in data preparation, ASKE requires to specify a set of seed concepts. A seed concept can be expressed as a short text (e.g., one or two phrases) providing a gross-grained description of the target. In this case, as a common example, a seed concept can be specified by taking an excerpt from pertinent law/case law documentation. As an alternative, a seed concept can be defined as a list of keywords. As an example, for a seed concept about banking contract, a corresponding list of keywords could be the following: bank deposit, safe deposit box, bank credit opening, bank advance, bank account, bank discount. Document chunk classification has the goal to annotate chunks with featuring concepts and zero-shot learning techniques are employed to this end. Zero-shot learning is an unsupervised classification technique, characterized by the ability to work without requiring any pre-existing annotation of the considered documents. Given a set of concepts (i.e., the seed concepts at the beginning of the process), a similarity measure , e.g., cosine similarity, is calculated over any pair of embeddings between chunks and concepts. A chunk is classified with the concept when the similarity value satisfies (k, c) ≥ , with defined as a similarity threshold configured in the system. A concept in ASKE is defined as a pair = (, )¯ , where is a label featuring the meaning of the concept expressed in a synthetic and human-understandable way, and ¯ is a vector-based concept representation. Each concept is initially associated with the set of terms extracted from the textual description of . The vector concept ¯ is built as the mean of the vectors of all the terms in . The label corresponds to the label of the term ∈ , whose vector representation ¯ is closest to the concept vector ¯. Terminological enrichment is then enforced to enrich the term set of a concept by considering the terms of any chunk classified with . The idea is that the initial description of the concept can become more detailed if we add terminology taken from chunks that are pertinent (i.e., classified) with . This is done by calculating the similarity between any pair of embeddings ¯ and ¯ in and . The most similar terms of are inserted in according to a system-defined similarity threshold.
Concept derivation is finally executed to determine new and more fine-grained concepts that can emerge from existing ones after enrichment. Given a concept , the Afinity Propagation (AP) algorithm is employed to cluster the embedding vectors ¯ of terms in . A new concept ′ is created for each cluster returned by AP. A link is defined between a concept ′ and to denote that ′ is derived from and they are somehow similar/related in content. The concept is then updated since the terms in can be changed due to enrichment. As a consequence, and ¯ are re-calculated.
ASKE endpoint. The set of concepts obtained after derivation can trigger the execution of a new cycle based on the above three steps. Each cycle execution is called ASKE generation. The new concepts derived in a certain ASKE generation contribute to improve the classification of chunks in more fine-grained concepts. New concepts can also be discovered through a new execution of enrichment and derivation on the basis of a refined classification result. As such, concept extraction terminates when the number of new concepts created in the derivation step is lower than a predefined termination threshold. A final concept graph ACG is populated with all the concepts and corresponding derivation links extracted by ASKE.
To show ASKE in action, we consider a case study of legal knowledge extraction from a legal corpus in the context of the NGUPP project. The case study dataset is composed by 50 court decisions from several Italian courts, spanning from year 2008 to year 2022. All decisions are ifrst-degree verdicts selected by legal experts of the project for their relevance to the subject matter unfair competition. We illustrate knowledge extraction from the unfair competition dataset by considering two use cases, modeling the use of ASKE by two categories of users with diferent levels of expertise, namely, UC-A, related to a legal practitioner user (e.g., a lawyer) with high legal expertise, and UC-B, related to a general subject (e.g., a citizen) with limited legal expertise. To trigger the ASKE extraction process, user A provides a legal definition as a seed concept, i.e., the expression “acts likely to cause confusion”, taken from art. 2598 of the Italian Civil Code. In the second case, user B provides a seed in form of general keywords like “bag, distinctive elements, imitation”. The seeds were in Italian, and translated here in English for readability. We asked two legal experts to evaluate the knowledge output extracted by ASKE in both use cases. In particular, we asked to qualitatively assess i) the pertinence of discovered chunks with respect to the initial seed concept, and ii) the appropriateness of the new ASKE concepts derived from the seed. In both cases, ASKE was running with hyperparameters = = 0.3, number of generations equal to 21, paraphrase-multilingual-MiniLM-L12-v22 as the embedding model and Open Multilingual WordNet as external dictionary to retrieve term definitions 3.
positively noted that concepts derived from the seed are coherent with the topic of the law
provision. Indeed, concepts like exemplify and relevant are related to proof of the uniqueness of
a product, while hindrance can be interpreted as a consequence of the acts producing confusion.
Concept clarification is related to exemplify, and other derivations, such as that of prudence,
debate, caution and dispute from relevant, though not strictly related from a semantic point
of view, were judged as appropriate in this context, since relevance is a key feature in legal
debate and disputes, while prudence and caution are exercised to prevent the introduction of
misleading, irrelevant or prejudicial information and to evaluate the reliability of evidence.
2https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
3Further details about the ASKE configuration in this case study are provided in [
Next, legal experts analyzed the chunks with the highest similarity with respect to the seed concept. We report below the top-two similar chunks as an example:
Suitability to cause confusion, therefore, consists of two elements: 1) the originality of the imitated product, endowed with distinctive capacity, such as to become inherent, in the image with the consumer, of the product itself; 2) the absence of distinctive elements capable of showing that the origin of one product is diferent from that of the other. [...] b) conversely, infringement only exists where there is a likelihood of confusion for the public, consisting even in a mere danger of association between the distinctive elements. Prerequisites for the aforementioned discipline to operate, therefore, are: i) the existence of substantial identity or similarity between the signs; ii) their use for goods and services that belong to the same sector and are intended to satisfy the same market requirements; iii) identity of characteristics in the eyes of the same average consumer or only relative afinity.
Both chunks are definitions of the conditions that characterize the “ acts likely to cause confusion”. These and other similar chunks identified by ASKE can therefore be exploited by legal practitioners for interpretation of a new case. Indeed, their main aim is to verify that a certain fact fits or doesn’t fit some conditions established by the law.
Figure 3 shows a portion of the concept graph produced by ASKE in case UC-B. In this graph legal experts noticed the prevalence of general concepts over legal ones. Two concepts emerge among others, related to the two senses of the Italian word “borsa”: “bag” and “stock exchange”. Looking at concepts derived from these ones, it can be noticed that ASKE was able to perform a correct distinction between the two.
Looking at the document chunks with the highest similarity with respect to the seed concept, we highlight two chunks that refer to each of the derived concepts.
As noted in the aforementioned judgment No. 5443/2017, “in the present case, such reproduction also applies to details such as, for example, the slightly rounded flap situated between the two handles and covering part of the zip fastener which, if they constitute an integral part of the shape of the bag model, nevertheless also appear to be elements in themselves capable of impressing themselves on the mind of the consumer who will be able to distinguish between products even legitimately having similar shapes, the one attributable to the source of production constituted by the present plaintifs.” Therefore, S.’s clients who intend to make investments of a financial nature first enter into a so-called ’placement contract’ with S. itself, and then enter into the actual contracts relating to their investment (subscription of units of mutual investment funds, or shares in SICAVs, or conclusion of an insurance policy, or conclusion of a portfolio management contract) directly with the ’product companies’ contracted with the plaintif.
The retrieved chunks were evaluated positively by legal experts from the point of view of their pertinence to the seed. Also the relatedness of extracted concepts starting from initial seed has been evaluated as satisfactory. The application of ASKE for the exploration of a legal corpus proved promising, as the concept graph simplifies the navigation of the underlying corpus making it accessible to even non-expert users.
With the recent progress of digital transformation, increasing interest and eforts have been
devoted to the development of advanced, AI-based approaches to process huge volumes of
legal digital documents and extract knowledge from them. In [
It is worth noting that ASKE enforces document classification without the need to rely on preexisting annotations, and without requiring to pre-define the number of target topics/concepts to discover. Thus, ASKE is particularly appropriate to satisfy exploratory information needs in those situations (e.g., the legal domain) where a-priori knowledge about the corpus is not available.
We also note that a key component in ASKE is Sentence-BERT [
In the paper, we presented the ASKE approach to legal knowledge extraction, which is based on
a combination of context-aware embedding models and zero-shot learning techniques. A deep
evaluation of ASKE has been performed considering the EurLex dataset [15] containing 45, 000
EU legislative documents in English, each of which is annotated by the Publication Ofice of
the EU with one or more labels from the EuroVoc thesaurus4. The goal of the evaluation was
twofold: i) to assess the quality of the knowledge extraction process, by assessing the capability
of ASKE to reconstruct the EuroVoc labels as extracted concepts, and ii) to evaluate the quality
of the document classification process, by assessing the correctness of ASKE concepts assigned
to each document against the ground truth labels. The results of the evaluation are positive on
both sides (see [
This work was supported in part by project SERICS (PE00000014) under the NRRP MUR program funded by the EU - NGEU. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the Italian MUR. Neither the European Union nor the Italian MUR can be held responsible for them. s10506-021-09301-8. [15] I. Chalkidis, E. Fergadiotis, P. Malakasiotis, I. Androutsopoulos, Large-scale multilabel text classification on EU legislation, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 6314–6322. URL: https://aclanthology.org/P19-1636. doi:10.18653/v1/P19-1636. [16] V. Bellandi, S. Castano, S. Montanelli, D. Riva, et al., A service architecture for ai-based legal knowledge extraction, in: CEUR WORKSHOP PROCEEDINGS, volume 3478, CEUR Workshop Proceedings, 2023, pp. 110–119.