Extracting information from court process documents to populate a knowledge base produces data valuable to legal faculties, publishers and law rms. A challenge lies in the fact that the relevant information is interdependent and structured by numerous semantic constraints of the legal domain. Ignoring these dependencies leads to inferior solutions. Hence, the objective of this paper is to demonstrate how the extraction pipeline can be improved by the use of probabilistic soft logic rules that re ect both legal and linguistic knowledge. We propose a probabilistic rule model for the overall extraction pipeline, which enables to both map dependencies between local extraction models and to integrate additional domain knowledge in the form of logical constraints. We evaluate the performance of the model on a German court sentences corpus.
In the year 2018 alone, there were approximately 870,000 court procedures in
Germany4. All of them are documented in text, however to this date there are
still only rudimentary solutions as to how to search for information within these
documents. Transforming these documents into structured form produces data
that can give insight into court processes and may be a valuable source to
examine research questions such as the perceived imbalance in degrees of penalty
between di erent regions in Germany, as described in
Each document results in one entry in a knowledge base with seven facts about the document speci c case and defendant, delineated in Section 3. Figure 1 displays the extraction pipeline to populate such a knowledge base and visualizes the dependencies between local components. It consists of the following two stages: 1. Document Classi cation: classi cation on document level predicts database entries with a xed set of possible values, such as the type of court sentence ( ne, imprisonment or acquittal). 2. Named Entity Recognition (NER): classi cation on token basis extracts all information with an unlimited set of possible values that have to be extracted from the document text itself, such as the duration of imprisonment. This will be treated as a named entity recognition task with the objective to assign a xed set of classes to each token and extract the information from the token text that is assigned the respective entity class. If a model assigns the same class to multiple text spans, they have to be consolidated to extract one value per document.
This baseline approach introduced for the use case raises multiple challenges: error propagation in the pipeline, value consolidation of NER predictions and the sparsity of training data.
Error propagation: One major disadvantage of this approach is that errors
are propagated from the rst classi cation component on document basis to
the second classi cation component on token basis. This may lead to a lower
performance with respect to the nal aggregated document results. This problem
is known from other NLP tasks such as knowledge base population. Recent work
therefore focused on solving multiple tasks and modeling it as an end-to-end
statistical inference problem
Value consolidation: One additional source of error is the consolidation of multiple token spans in the same document to extract one value per document. A NER classi er may in one document classify multiple token spans with the same class and as a consequence create multiple candidates that need to be consolidated to generate one valid value per NER class and document. Assigning the same NER class to multiple spans in one document is only valid when they refer to the same entity, e.g. the text span is similar. Integrating this as a hard constraint in a post-processing step may introduce additional errors.
Label sparsity: Since obtaining, anonymizing and labeling court sentence documents is a tedious process, the corpus available to train the pipeline classiers is rather small. Beside the raw data there is valuable knowledge, both about dependencies between the information (e.g., a defendant with previous convictions is more likely to get imprisoned) and about the document structure (e.g., date and location of the court are likely to be in the beginning of the document). This sort of knowledge is not exploited in traditional approaches.
In order to address these challenges, we propose a probabilistic model constructed with a logical templating language that models a joint objective over the whole pipeline. By reasoning over all pipeline tasks jointly, we try to weaken the e ect of propagated errors. By then modeling dependencies between NER candidates per document we address value consolidation challenges and with using a logical templating language, we enable to integrate additional background knowledge. Contributions of this paper can be summed up as the following: { We propose a probabilistic model based on PSL for the overall extraction pipeline, which enables to map dependencies between local classi cation models. { We model relevant concepts from the legal domain in the form of logical constraints and integrate them into the probabilistic model. { We provide an empirical evaluation of the approach on a data set of court sentences and compare results with the traditional pipeline approach. The remainder of this paper is organized as follows: Section 2 gives an overview of related work in the eld of joint information extraction and probabilistic pipelines, Section 3 describes our approach, Section 4 provides an experimental evaluation on a German sentencing corpus and reviews benchmarks in comparison with a traditional pipeline approach, Section 5 concludes with a summary and future work. 2
Modeling pipelines in NLP and handling error propagation is a well known
problem in multiple elds of research. Former work proposed diverse approaches
to tackle this, e.g. by using graphical models or inductive logic programming.
We propose a probabilistic model for an information extraction pipeline that
jointly reasons over two pipeline stages: the Document Classi cation Stage
and the NER Stage. This model is constructed using the logical templating
language Probabilistic Soft Logic introduced by
In this stage, all defendant and case-related information such as the type of court
sentence is extracted. The objective of this stage is to perform three separate
classi cation tasks and assign the following categories to each document:
{ Previous Convictions (PC) : Yes/No
{ Type of court sentence6 (TS): Imprisonment/Fine/Acquittal
{ Probation (PR): Yes/No
For each classi cation task, two separate model architectures are trained and
applied, the inbuilt convolutional neural network (CNN) based model provided
by spaCy 7, as well as a transformer based classi cation model, BERT
In the NER stage all document-class speci c information such as amount of the ne have to be extracted from the document. Objective of the NER stage is to assign one of the following classes to each token in the document: date of the court sentence (date), court location (loc), amount of the ne (f amnt), number of day- nes (f units)10 and duration of imprisonment (d impr). Whether f amnt and f units or d impr are present in a document depends on the type of court sentence (imprisonment or ne). For this classi cation task we rely on two architectures described below.
BERT NER: The named entity recognition model based on BERT has the same general transformer architecture as the classi er. Instead of predicting the class of sentences, a per-token classi cation takes place. We use a single feedforward layer with softmax activation to calculate the token labels. spaCy NER: The spaCy NER model11 utilizes a di erent architecture. Its embedding layer consists of two parts incorporating syntactic knowledge about the word, and a stack of 4 residual CNN layers to capture contextual information. Both layers are followed by a feed-forward network. Finally, a form of attention mechanism is used to incorporate additional information about the previous tokens and previous entity predictions. The nal layer is a feed-forward and correction layer, which prevents illegal state shifts 3.3
We introduce a graphical model of a joint probability distribution to reason
over all stage outputs at the same time. This model integrates both local stage
model predictions, dependencies between the stages and additional background
knowledge. For model construction we use the Statistical Relational Learning
Framework PSL, introduced by
P (Y jX) = 10 In German criminal law, nes are calculated in day- nes. The number of day- nes depends on the severeness of the o ense while the amount of each day- ne is based on the o ender's personal income. 11 https://spacy.io/universe/project/video-spacys-ner-model with Z as a normalization factor.
j (X; Y ) = (maxflj (X; Y ); 0g)pj (2) with lj representing a linear function and pj 2 f1; 2g.
Potential functions j are de ned per clique, a subset of fully connected nodes in the graph, assigning a probability mass to each clique state. A clique state is one assignment of values to all random variables participating in the clique. Assignment of a higher value to one clique state means that this state will be interpreted as being more likely. In PSL, these potential functions are generated using logical rst order rules, such as: w : F riends(A; B) ^ F riends(B; C) ) F riends(A; C) (3) A weight w is assigned to each rule and indicates its importance. F riends is called a predicate. Predicates can take one to multiple arguments, such as A and B. Both A and B are variables and serve as placeholders. They can be substituted by concrete instances, referred to as constants. A PSL-model itself consists of a set of these template rules and a weight for each. Substituting all variables in the rule template set with their respective constants is called grounding. Every grounded rule represents one clique in the underlying graph structure and every grounded predicate is an observed or unobserved random variable mapped to a node in the clique. The weight of a grounded rule determines the weight !j of a potential function. A potential function assignment denotes the degree to which a rule is satis ed. Clique states that lead to satisfying a rule will be assigned a higher value than clique states that lead to the violation of a rule. Intuitively spoken, assignments of Y are more likely the fewer rules they violate.
Our proposed rule set can be categorized into four rule types: basic rules, domain rules, pipeline rules and similarity rules, examined in the following subsections. Table 1 provides a detailed explanation for all elements participating in each rule.
Basic Rule Set: The basic set models the relationship between the local classi cation models (re ected by the observed predicate f cls pc(s1) for stage one and f cls(s2) by stage two respectively) and the true unobserved stage prediction (cls pc(s1) for stage one and cls(s2) for stage two).
f cls pc(s1)(d; m; c pc) ^ T (m; c pc) ) cls pc(s1)(d; c pc)2 !f cls pc(s1)(d; m; c pc) ^ T (m; c pc) )!cls pc(s1)(d; c pc)2 f cls(s2)(d; z; m; c) ^ T (m; c) ) cls(s2)(d; z; c)2 !f cls(s2)(d; z; m; c) ^ T (m; c) )!cls(s2)(d; z; c)2 cls pc(s1)(d; +c pc) = 1: !cls(s2)(d; z; c) (4) (5) (6) (7) (8) d = f1; :::; Dg with D as the number of documents in the corpus m = fSP ACY; BERT g local stage model z = f1; :::; Zg with Z as the number of token candidates c pc = fyes; nog class types predicted by P reviousConviction classi er c st = fF ine; Imprisonm:; Acq:g class types predicted by courtsentencetype classi er c pr = fyes; no; otherg class types predicted by probation classi er c = fdate; :::; locg class types predicted by N ER classi er Observed Predicates f cls pc(s1)(d; m; c pc) f cls st(s1)(d; m; c st) f cls pr(s1)(d; m; c pr) f cls(s2)(d; z; m; c) CLOSE(d; z1; z2) SIM (d; z1; z2) prediction output of the local P reviousConviction classi er model m for document d and class c pc prediction output of the local courtsentencetype classi er model m for document d and class c st prediction output of the local probation classi er model m for document d and class c pr prediction output of the local model m for document d, token candidate z and class c assigned 1, when there are at most 20 token between z1 and z2, 0 otherwise assigned 1, when the text z1 and z2 span over is equal, 0 otherwise Unobserved Predicates cls pc(s1)(d; c pc) cls ts(s1)(d; c ts) cls pr(s1)(d; c pr) cls(s2)(d; z; c)
Rules 4 and 5 display the rules for the local classi cation models from stage one
for P reviousConvictions. T (m; c) is a trust score introduced by
Figure 2 displays a simpli ed model of the grounded observed (grey nodes) and unobserved predicates (white nodes) according to the above rule sets. The model is grounded only for one class (c1 pv, c1 pr) of the document classi ers pc and pr, for one document d1, two local stage one models m1 and m2, two local stage two models m3 and m4 to predict one class c3 and two NER candidates z1 and z2 for which class c3 should be predicted. Potential functions resulting from the baseline rules are presented as red squares.
Pipeline Rule Set: This rule set incorporates relationships between prediction outputs of the document and NER classi cation stages and relationships between local models from one stage. Rule 10 for example expresses the dependency between the st classi er and the NER classi er. Whenever the predicted value for court sentence type is not " ne", it is unlikely that the information about the amount of the ne is present in the document
Rule 11 models the dependency between the P R and the ST classi ers for of the rst stage.
!cls st(s1)(d;0 f ine0) )!cls(s2)(d; z;0 f amount0)2 cls pr(s1)(d;0 no0) ) cls st(s1)(d;0 imprisonment0)2 Domain Rule Set: The domain rules permit to incorporate knowledge about the document structure (e.g. in Rule 12) and class dependencies such as that defendants with previous convictions are more likely to be sentenced to imprisonment or that the height of the ne and the number of day- nes are most likely mentioned in the same paragraph (see Rule 13).
cls pc(s1)(d;0 yes0) ) cls st(s1)(d;0 imprisonment0)2 cls(s2)(d; z1;0 f amnt0)^!CLOSE(d; z1; z2) ) cls(s2)(d; z1;0 f units0)2 (10) (11) (12) Similarity Rule Set: The similarity rules model the relationship between multiple mentions of the same entity in one document. Rule 14 denotes that when two candidates in one document span over similar strings, they are more likely to be assigned the same NER type. This ensures that when one information like the date of the court sentence is mentioned multiple times in the document it is always assigned the same NER class type and when two candidates don't have equal content, they are unlikely to refer to the same entity (Rule 15). cls(s2)(d; z1; c) ^ SIM (d; z1; z2) ) cls(s2)(d; z2; c)2 cls(s2)(d; z1; c)^!SIM (d; z1; z2) )!cls(s2)(d; z2; c)2 (14) (15) 4
Data: We evaluate the performance of the model introduced in Section 3 using a corpus with 146 German court sentences12. Each document describes an o ence (theft), where the defendant is either sentenced to imprisonment or to a ne, and which is only related to one case and one defendant. The documents are split into training, validation and test set.
PSL Pipeline Model: For the probabilistic model we train local stage
classiers for document and token classi cation using the architectures described in
Section 3. All classi ers are trained on the training set and applied to the
validation and test set. We use the validation set to estimate the trustworthiness
scores (Rules 4, 5, 6, 7) and to re ne weights. To evaluate the performance of the
rule model and its subsets, we do not provide an evaluation for the local stage
tasks (document and token classi cation), but focus on the amount of correctly
extracted values per document. We perform inference for the unobserved
variables listed in Table 1 to predict all layer outputs at the same time. As proposed
in
Basic Pipeline Model: We compare the proposed solution to the pipeline approach visualized in Figure 1, where all local models are trained using the BERT architecture (see Section 3). Since the models that were trained using the architecture of spaCy had a poor performance on the test set, the performance of the pipeline model based on these classi ers is not evaluated here. The local document classi cation models are trained on both training and validation set and are applied to the test set to predict document class labels. The local token classi cation models are trained both on training and validation set and are then applied to the test set. The N ER classi cation results per document are consolidated by selecting the candidate with the highest probability score predicted by the model for each N ER class. We model dependencies between the 12 manually annotated by the University of Cologne predicted type of sentence and the N ER prediction as hard constraints. When the predicted class is "Fine", prediction for "f amnt" and "f units" are set to "NaN" (for document class "Imprisonment", prediction for "d impr" is set to "NaN" respectively). We additionally provide results for the extraction workow where this stage dependency is not modeled and contradicting results are not removed. Table 2 provides a comparison of model performances. The rst seven entries denote the performance of the rule models consisting of di erent rule subset: basic (basic rules), sim (similarity rules), pipe (pipeline rules) and domain (domain rules). The scores per extracted information type re ect the amount of correctly extracted values on the test corpus.
The two PSL models achieving the highest overall matching rate both contain basic, pipeline and domain rules. Modeling relations between N ER candidates by integrating the similarity rules does not seem to have an in uence on the performance in this experiment. On average over all information types, the highest scoring PSL model outperforms both BERT benchmark models. For some particular types (Date, d impr and st) the BERT benchmark model achieves a higher matching rate. These results indicate that modeling the pipeline in a probabilistic way and integrating domain knowledge improves the overall performance. Since the data set is very small, further analysis on a bigger corpus and a more detailed analysis of the rule model e ects are required. 5
This paper proposes a rule based probabilistic model to improve the extraction pipeline for information extraction from court sentence documents. The model enables to both map dependencies between local extraction components and to integrate additional domain knowledge in the form of logical constraints. We evaluate the performance of the model on a German court sentences corpus and show that the model improves results compared to a BERT benchmark model. 6
This research has been funded by the Federal Ministry of Education and Research of Germany as part of the competence center for machine learning ML2R (01IS18038B)