In this study, we analyze data-scarce classification scenarios, where available labeled legal data is small and imbalanced, potentially hurting the quality of the results. We focused on two finetuning objectives; SetFit (Sentence Transformer Finetuning), a contrastive learning setup, and a vanilla finetuning setup on a legal provision classification task. Additionally, we compare the features that are extracted with LIME (Local Interpretable Model-agnostic Explanations) to see which particular features contributed to the model's classification decisions. The results show that a contrastive setup with SetFit performed better than vanilla finetuning while using a fraction of the training samples. LIME results show that the contrastive learning approach helps boost both positive and negative features which are legally informative and contribute to the classification results. Thus a model finetuned with a contrastive objective seems to base its decisions more confidently on legally informative features.
is causing a bottleneck in legal text classification
research [
While there are a few publicly available
datasets, such as CUAD [
tive solution for classification tasks with small datasets where data annotation is also time-consuming, ineficient and expensive. These methods are designed to work with a small number of labeled training samples and typically cific downstream task.
In this paper 1, we focus on three major aims. First, we
ifnetune the LegalBERT[
ISN1613-073 https://www.shareforcelegal.com (B. Kilic); https://www.uu.nl/staff/FJBex (F. Bex); https://albertgatt.github.io/ © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop Proceedings (CEUR-WS.org)
need” [
Secondly, we finetune the same baseline model with
these two finetuning objectives with the balanced dataset
created from LEDGAR. Lastly, to analyze the
trustworthiness and explain individual predictions, we extract the
tokens from the model as features by using LIME [
ifnetuning setups that can potentially address this necessity, such as parameter eficient finetuning (PEFT), pattern exploiting training (PET), and SetFit (Sentence
framework for few-shot finetuning of Sentence Transformers (ST). SetFit works by first finetuning a pretrained
of ST, addressing labeled data scarcity by introducing contrastive learning methods to generate positive and negative pairs from the existing dataset to increase the number of samples.
tence embedding. Next, the classification head that was trained in the training step, produces the class prediction of the input sentence based on its sentence embedding. Formally this is predicted label = ( ()) , where
represents the classification head prediction function.
LEDGAR dataset, and on a balanced version. We describe the original dataset first, then we give a brief description of how the dataset was further balanced for the presented the unseen inputs and produces the embeddings accord- experiments. ingly. Then the classifier head gives the prediction results based on the newly generated embeddings.
(, , 1) ST finetuning
To better handle the limited amount of labeled training data in few-shot scenarios, contrastive training approach is used. Formally, we assume a small set of K-labeled samples = ( , ), where and are sentences and their class labels, respectively. For each class label ∈ , positive triplets are generated: = , where and are pairs of randomly chosen sentences from the same class , such that = = . Similarly, a set of negative triplets are also generated: = (, , 0)
, where are sentences from class and
are randomly chosen sentences from diferent classes
such that = and ≠ . Finally, the contrastive
ifnetuning data set is produced by concatenating the
positive and negative triplets across all classes where ||
is the number of class labels, | | = 2||
is the number of
pairs in and is a hyperparameter. SetFit will generate
positive and negative samples randomly from the training
set, unless they are explicitly given [
This contrastive finetuning approach enlarges the size of training data. Assuming that a small number ( ) of labeled samples is given for a binary classification task, the potential size of the ST finetuning set is derived from the number of unique sentence pairs that can be generated, namely ( − 1)/2 larger than just .
Classification head training In this second step, the ifne-tuned ST encodes the original labeled training data { }, yielding a single sentence embedding per training sample: ( ) = ( ) where () is the function representing the fine-tuned ST. The embeddings, along with their class labels, constitute the training set for the classiifcation head = (( ), ) where | | = || for the vanilla finetuning setup. Therefore the maximum number of samples is calculated as: maximum number of samples per label multiplied by the number of total labels, as can be seen in Figure 1. In practice, in the case of the vanilla finetuning setup, we end up with fewer training samples than this total. This is because some labels are extremely sparse, and there are fewer total samples than the stipulated maximum per label.
est label consists of only 23 samples, and the largest has 3167 samples in the original training dataset. Therefore, to create a new balanced dataset, we selected the most frequent 32 labels.
For labels with more than 1000 samples, we downsampled to 1000 samples per label. For labels with fewer than 1000 samples, we upsampled by crawling and retrieving additional data from LawInsider,3 removing any duplicates. As a result, a new dataset has been created that consists of 32 classes, with each having 1000 provisions. , which is significantly
codes an unseen input sentence ( ) and produces a sen- 3https://www.lawinsider.com/
for the balanced data scenario, and selected 25 samples per label randomly for the 32 labels, for a total of 800 samples. The remaining 31,200 samples are used for training with a random 80/20 train/dev split.
For finetuning with the balanced dataset, we again train with varying sizes of training data, using 4, 8, 12, and 16 samples per label for SetFit, and 50, 100, 150, and 200 for the vanilla finetuning setup, as can be seen in Figure 2.
Note that, unlike the case of the unbalanced data, the total sizes for the vanilla finetuning setup in the balanced case correspond to the totals obtained by multiplying the maximum sample size with the number of labels.
It has been shown that models which have been
pretrained on domain-specific legal data outperform
generalpurpose models [
periments, with the test set described in Section 3. The
We observed that the models achieve comparable accuracies overall, despite the diferences in Weighted F1scores in the Table 2. However, it is still noteworthy that the contrastive learning approach achieves accuracy comparable to the vanilla finetuned model with very small sample sizes.
In Figure 2, we compare the accuracy of the two approaches, this time with the balanced LEDGAR dataset.
In this experiment we also used 80/20 train/dev split. The results show that the contrastive learning finetuning has a warmer start compared to vanilla finetuning, particu- sections, we observed that SetFit models were compalarly in small data scenarios. However, as can be seen rable with vanilla models, despite using a fraction of from the graph, SetFit is comparable with vanilla model the dataset. However, we get very little information across all the experiments as well. about whether the models base their decisions on features which are intuitively correct, that is, if the models 5.3. LIME feature comparisons are classifying the provisions with legally informative features, or arbitrary ones.
In machine learning in general, but especially in domains LIME is a technique based on the creation of
intersuch as law, trustworthiness of AI systems is crucial. pretable, surrogate models over the features that are
loThe ability to explain model predictions is central to cally faithful to the original classifier. This means that
increasing trustworthiness in at least two respects. First, interpretable explanations need to use representations of
explanations have an impact on whether a user can trust those features that are understandable, trustworthy, and
the prediction of the model to act upon it; second, they justifiable to humans [
Several approaches to explaining model predictions Thus, the positively weighted words that lead toward a
have been proposed in the literature, including LIME [
Through the training results mentioned in previous timate of the probability of the label are called ”negative” Table 3 like ”authority”, ”power”, ”act”, ”execute”, ”binding”, etc. F1-score comparisons of Adjustments and Authority provi- for the Authority provision. Thus, domain experts can sions make decisions based on their usefulness.
Class Label Vanilla SetFit In Figures 4 to 7, we show the top positive features for the two models separately, for each label. We note Adjustments 0.7368 0.8571 that similar observations can be made with respect to Authority 0.5063 0.2903 these figures, that is, the contrastive learning framework boosts the positive weight of features that are intuitively more legally informative. Nevertheless, we also see that features. less informative features, including stop words, are also
We kept the LIME hyperparameters the same in each assigned some positive weight. model explanation for fair comparison and the details are Additionally, we also observed similar behavior with as follows: The limit for the total number of words per the negative features in Figures 9 to 12. For negative classification is defined as , and the complexity measure features, the SetFit model trained with a contrastive obfor the models is defined as: jective assigns a greater negative magnitude. Thus, it Ω() = ∞[‖ ‖0 > ] appears that negative role of these features is accentuated in the contrastive setting, relative to the standard finewhere the is defined as a simple interpretable sparse tuning setup. For instance, words like ”changes”, ”shall” linear model (logistic regression in the case of SetFit, and ”without” for the Adjustments provision and ”which”, multinomial logistic regression in the vanilla model); ”common”, ”document” and ”carry” for the Authority prois defined as the weight vector of . The = 10 is vision sound generic and may not give legally informaselected across all the experiments for simplicity and tive hints to humans. However, in the vanilla model case, potentially can be as big as the computation allows. The similar legally non-informative negative features are also size of the neighborhood for local exploration is set to 25. present but not enough to perturb the model’s decisions. The distance function was kept as the default, cosine distance metric. 7
Thus, in this section, we compare the positive and 6. Conclusions & Future Work negative features of SetFit and vanilla models extracted using the LIME setup mentioned above. This paper presented a detailed comparison of legal pro
To ensure a fair comparison, we used the SetFit model vision classification. Motivated by the challenge of
lowtrained with 800 training samples and the vanilla model resource scenarios and data imbalance, we compared
trained with 9756 training samples. As shown in Figure 1, the performance of a LegalBERT model finetuned in a
the two models converged and obtained comparable per- standard setting, to one finetuned using a contrastive
formance with these settings. We selected two test labels objective.
to compare, namely Adjustments and Authority provi- Following previous work [
We begin by comparing the positive features which tracted from the models difer significantly, favoring the both approaches have in common (i.e. the features they SetFit model, despite using almost 11 times less data. both assign a positive weight to), for the two target la- As future work, investigating the limitations of SetFit bels. These are shown in Figure 3 and Figure 8. The deeper with more hyperparameters on legal data may be ifgures suggest that the contrastive approach from SetFit beneficial for pushing the model capabilities further. Also, seems to help to boost legally informative features more we plan to use other explainability tools such as SHAP or than vanilla models, even in the small data scenarios. For GRAD-CAM to compare the extracted features. Finally, instance, words like ”adjustments”, ”shares”, ”dividend”, an evaluation of the appropriateness of the positive and ”stock”, etc. can give a first strong hint about the Adjust- negative features identified using explainability methods ment provision classification results, as well as words needs to be carried out with domain experts.