In this paper we examine the ability of pre-trained language models to generalize beyond the legal domain and dataset they were trained on. We transform three separate datasets of annotated legal cases with different type systems to use a single type system (namely, determining whether a sentence describes facts or not). Then, we train SVM and RoBERTa models on the different source datasets, and measure their performance on the remaining datasets. We also create models trained on combined datasets, and measure their performance. Overall, we find that the performance of the models is surprisingly high, despite being trained and evaluated between datasets from different domains and jurisdictions. The RoBERTa model seems to be able to benefit from its pre-trained language representation to better capture the abstract idea behind the new type system. Further, the models benefit from training on multiple datasets. This potentially opens the door for models trained on separate datasets that could perform better in previously unseen domains, and to overcome cold-start problems when using active learning methods.
In this paper we examine the ability of pre-trained language models to generalize beyond the domain and dataset they were trained on in the legal domain. We explore if and how different datasets and type systems can be re-cast in such a way that a language model trained on one dataset could be successfully utilized on other datasets. As highquality legal data sets are scarce, the efficient use of those that are available is of utmost importance in AI & Law research.
Written legal cases typically follow a certain pattern in their reasoning. For example, legal cases often contain the following sections:
1. Claims of the plaintiff and responses/counter-claims of the defendant 2. A description of factual circumstances of the case as seen by the parties and as determined by the court 3. Legal rules applicable to the factual circumstances 4. Application of the rules to the factual circumstances 5. Court’s conclusions and outcomes of the case The sections play different roles in a decision, and could carry different meaning dependent on the context in which they are read. A judge, a law student, a party to a dispute, or a legal professional could all read the different sections with different focus depending on what they need to learn. It is an indispensable skill for any professional lawyer to quickly identify and understand the different sections of a case.
Due to its prominence the task has attracted abundant research in AI & Law (see
e.g. [
Typically, research in prediction of rhetorical roles includes dataset creation and training of ML models on that dataset. The dataset usually focuses on one or several domains. The model is then tested against a test set, typically sampled randomly from the dataset. This shows the predictive capability of the model on test data from the same distribution as the training data.
However, it can be difficult to evaluate how well a model would perform if applied to other, previously unseen datasets that might have different distributions, e.g. whether the model is able to generalize beyond the annotated dataset. For example, it is possible that a model has only learned a specific characteristic of a single domain or jurisdiction to describe factual circumstances, rather than the abstract idea of what a sentence describing factual circumstances looks like. If the goal is to apply the model in a general fashion, on cases from different jurisdictions, domains and countries, it is therefore important to evaluate the performance of the system on datasets it has not seen at all during training.
Evaluating the models on multiple datasets, however, is not trivial. Firstly, publicly available datasets are scarce in AI & Law. Further, the datasets that are available typically use custom type systems, making cross-dataset evaluation challenging. In this paper, we identify an example meta type system, that generalizes across three datasets. We use the meta type system to train models on data from different domains and jurisdictions and assess performance of the models across the domains.
We compare the ability of different types of ML models to generalize from one dataset to another. First, as a baseline we use a Support Vector Machines classifier which learns the correlation between word and/or n-gram occurrences and a certain label. Then, we evaluate the more recently developed BERT model (bidirectional encoder representation from transformers). This model has been pre-trained on massive corpora of text, to learn a language model. It can then be further fine-tuned on a down-stream task, such as text classification in a specific domain, achieving strong results with little training data by leveraging the previously learned language representation.
By conducting the experiments described in Section 4, we investigate the following hypotheses: • H1 - Machine learning models are able to generalize the knowledge learned from one dataset and apply it successfully on another dataset. • H2 - The BERT model is able to leverage its understanding of language to abstract beyond the specific domain vocabulary, thereby showing better performance in generalizing across the domains than the SVM model. • H3 - Training the models on pooled data from multiple datasets improves performance and results in more robust models.
Showing that the models generalize well among different domains could have important implications for the real-world applicability of the methods. Further, they could be employed in active or interactive learning settings, where annotators are presented with predictions by the model and confirm/correct these to create accurate models. This could drastically lower the number of expensive annotations that need to be performed to start the annotation of novel datasets.
The identification of rhetorical rules of sentences in legal cases has been investigated by
several researchers. The authors of [
We further investigate whether BERT-based models are able to exploit their
pretrained language model to achieve higher performance in generalizing across datasets.
The use of BERT on legal texts has numerous examples. In [
We show that models pre-trained on one dataset can to some extent perform
predictions on other datasets. This could be used to bootstrap new datasets in other domains,
decreasing the effort of annotation. This follows a steady line of work in AI & Law
on making annotation more effective. Westermann et al. [
In order to evaluate the ability of models to generalize beyond a single domain, we
employ an experimental design consisting of several steps. First, we identify three datasets
containing a categorization of sentences by rhetorical role (Section 4.1). Then, we
identify a meta type system that we can transform all the type systems into to create a task
shared among the datasets (Section 4.2). We then fine-tune and evaluate a pre-trained
language model and a Support Vector Machine model (Section 4.4) on different
combinations of these datasets (Sections 4.4 and 4.5).
4.1. Data
In this work we utilize three datasets. The first one comes from [
The second dataset also focuses on the rhetorical roles of sentences. Bhattacharya
et al. [
We also created a brand-new data set by scraping the case briefs from a publicly
available Case Brief Summary database (“CB”).4 The case briefs were categorized in
terms of the areas of regulation, such as administrative law, business law, or criminal
law (11 categories in total). In total, we were able to obtain 715 unique case briefs.
The case briefs are structured into a number of sections with headings. We extract the
sections based on an extensive battery of regular expressions to segment the retrieved
briefs into individual sections. While there were over 100 unique section heading names
we were able to identify six main types to which we could map many of the different
variations (e.g., all of Legal Issue, Issues, and Issue map to a single category). We applied
a specialized legal case sentence boundary detection system [
In order to map the source type systems into a single type system, we identify the label of the source datasets that most closely corresponds to a description of factual circumstances in the cases. These sentences are tagged as “Facts”. All other sentences are tagged as “Non-facts”. The labels that we chose to represent facts in the different datasets are as follows: 1. BVA - “Evidence” - defined as a sentence that primarily states the content of the testimony of a witness, states the content of documents introduced into evidence, or describes other evidence. 2. CB - “Facts” - Sentences stemming from a section with the title “Facts”, typically describing the factual background to a case. 3. ISC - “Facts” - refers to the chronology of events that led to filing the case, and how the case evolved over time in the legal system (e.g., First Information Report at a police station, filing an appeal to the Magistrate). 3Dataset available at github.com/Law-AI/semantic-segmentation 4http://www.casebriefsummary.com/; Note that as of 2020-11-29 the site is no longer available. After transforming the datasets into this schema, the distributions of Facts vs Non-Facts are as described in Table 1 for each dataset. While the source type systems are different, and the definitions of the corresponding types are not the same, for the purposes of the training and evaluation we assume that the new type system describes the same task when applied across the three datasets (i.e., we assume that we have transformed three related tasks into a unified one).
We randomly split all the data sets into training, validation, and test sets using ratios of 50-25-25. The splitting for all four datasets is performed at the level of documents, i.e., all the sentences from a single document are in the same fold. We did not take the size of a document into account, meaning that the number of sentences could vary slightly between datasets. The training split of the datasets is used for training the models, the validation split is used for model selection and hyperparameter optimization, while the test set is used for final evaluation.
In this work, we use the RoBERTa (a robustly optimized BERT pretraining approach)
described in [
As baseline, we use a Support Vector Machine (SVM) classifier. SVM constructs a hyper-plane in a high dimensional space, which is used to separate the classes from each other. As an implementation of SVM we use the scikit-learn’s Support Vector Classification module.6 As features we use the bag of words of (1-3)-grams weighted by TF-IDF.
We train the models on all the different possible pools of training data. The possible pools are BVA, CB, ISC, BVA+CB, BVA+ISC, CB+ISC, and BVA+CB+ISC. Both RoBERTa and SVM are trained on each of these pools separately, and the results are presented for prediction on the test split of each dataset (Section 5).
In all the experiments, we fine-tune the base RoBERTa model, with a linear classification layer on top of the pooled output, for 10 epochs on the training splits of the selected datasets. We use the batch size of 8 which is the maximum allowed by our hardware setup (1080Ti with 11GB) given we set the length of a sequence to 512 (maximum). As optimizer we use Adam with initial learning rate set to 4e 5. We store a model’s checkpoint after the end of each training epoch. The checkpoints are evaluated on the 5github.com/pytorch/fairseq/tree/master/examples/roberta 6scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html corresponding validation set. The model with the highest F1 on the validation set is then selected as the one to make predictions on the test sets. The performance on the test sets is what we report in the results section.
The SVM is optimized via gridsearch over the few most important hyperparameters (C, class weight, number of iterations). The various hyperparameter combinations are evaluated against the validation dataset. The model with the highest F1 on the validation set is then selected as the one to make predictions on the test sets. The performance on the test sets is what we report in the results section.
For evaluation we report the F1-measure (F1), i.e., the traditional information retrieval measures, to evaluate performance of the trained models. Since the task is binary, the application of the measure is straightforward.
jSj T P P = å i=1 T P + FP
jSj T P R = å i=1 T P + FN
F1 = 2PR P + R T P stands for true positives, FP for false positives, and FN for false negatives.
In order to evaluate this hypothesis, we investigate the scores in Table 2. Of course, the models with the highest scores are the models where the training data stems from the same datasets as the testing data (cells colored in gray). Further, the RoBERTa model performs better than the SVM model across the board, which is expected as RoBERTa is a much more complex model (compared to SVM) pre-trained for language understanding.
To investigate H1 we assess the scores for models trained on one dataset when applied an another dataset. For the RoBERTa model, the maximum drop in performance is 0.27 in F1-score, when trained on BVA and evaluated against ISC. In general, the drop is smaller, with strong performances even when applied on different datasets. The fact that the RoBERTa model trained on the CB dataset achieves and F1-score of .84 on the BVA dataset, despite the different characteristics of the datasets, is quite impressive.
The SVM model seems to have problems learning the distributions of the ISC dataset, resulting in low performance when trained on this dataset. This might be due to the transformed ISC dataset being more imbalanced than the other two (see Table 1. Overall, the RoBERTA model trained on the CB dataset performs the strongest on average. This might be due to the large distribution of domains contained in this dataset.
The RoBERTa model appears to transfer the knowledge learned on any one dataset to prediction on the other two much better than the baseline SVM model (Table 2). For example, RoBERTa trained on CB performs at F1 = 0:84 on BVA. This is a sizeable performance hit when compared to RoBERTa trained on BVA (F1 = 0:94). However, the SVM model goes from F1 = 0:92 to F1 = 0:60. While it is customary for BERT models to perform better than traditional ML models, the sizeable difference of .24 in this instance might indicate that there is a qualitatively different understanding going on, as RoBERTa can understand the semantic meaning rather than the specific vocabulary used. The difference does not hold up for all datasets, however.
Table 3 shows the results for models trained on the matching training data (e.g. the model trained on BVA evaluated on BVA) in the top row, and models trained on multiple pooled datasets and evaluated on a single test dataset below. Overall, there does not seem to be a strong improvement when adding another dataset to the training data.
Concerning robustness, the results suggest that RoBERTa is much better at handling data coming from different datasets, achieving impressive performances on both datasets. This is not the case for the SVM baseline. The RoBERTa model trained on all three datasets is especially impressive as it achieves within 0.01 of all the individual datasets simultaneously. Such a model would likely generalize better to other domains (not considered here) as compared to the model trained exclusively on one of the three datasets.
This robustness can further be illustrated by the fact that combining data from the two non-target datasets seems to yield higher performance on the target datasets than training just on either of the non-target datasets. For example, training the RoBERTa model on both BVA and ISC yields higher performance when predicting CB (.75) than using either BVA (.71) or ISC (.65) for training on their own.
The results reveal several interesting properties of the evaluated models. First of all, they show that it is indeed possible to train a model on one dataset, and use it for prediction on another dataset with decent performance. This holds especially true for the powerful RoBERTa model. It should be noted that the models in section 2 have never seen any sample from the target dataset, except when the source dataset and the target dataset are the same, and still achieve strong results. This shows a clear path towards using a model trained on one dataset to bootstrap a new dataset for active or interactive learning purposes. It also shows that the tasks defined in terms of their respective type systems are related, despite being created in different contexts and jurisdictions. The relatedness enables ML models to be successfully applied across domains.
Further, several interesting observations can be made when comparing the performance of the SVM and RoBERTa model. In several instances, the drop between the model trained and evaluated on the same dataset and a model trained on another dataset is larger for the SVM model. This could indicate that RoBERTa leverages its language understanding to abstract beyond the specific vocabulary used.
Finally, the RoBERTa models trained on multiple datasets seem to retain high performance on the specific datasets they are trained on, while also improving performance on datasets they were not trained on. This is another promising result showing that the models can learn patterns even from datasets created in different contexts and jurisdictions, and use the additional data to create robust and strong classifiers.
In this paper, we have provided an example of how datasets created in different contexts can be re-cast to support the same task, and utilized towards a common goal. The pretrained language models seem to be able to understand the underlying idea behind a task to a larger extent than SVM models, resulting in higher performance when applied to different datasets than they were trained on. Further, training the models on several domains improved robustness and to some extent performance. Overall, these results suggest that pre-trained language models have several desirable properties when training on multiple datasets, making them an ideal tool for efficient utilization of the existing AI & Law datasets to support future research in novel tasks.
We observed that it is possible to train a model on one dataset, and use it for prediction on another one with decent performance. We plan to investigate whether this could be used to overcome the cold start problem to improve efficacy in an active learning context. Another angle of investigation is to experiment with the transformation of the datasets, and identifying other pertinent documents, potentially even across languages. Finally, using larger, state-of-the art models is warranted to investigate additional improvements in predictive performance.