Legal case documents follow a common thematic structure with implicit sections like “Facts of the Case”, “Issues being discussed”, “Arguments given by the parties”, etc. These sections are popularly termed as "rhetoric roles". Knowledge of such semantic segments or roles will not only enhance the readability of the documents but also help in downstream tasks like computing document similarity, summarization, etc. The task is, given a legal document, to classify each sentence into 7 rhetoric roles. We compare ALBERT, BERT, RoBERTa and LaBSE for this task. The results show that BERT had the best accuracy at predicting the labels.
In AILA 2020, the task of labelling rhetoric roles for legal judgments was solved by many
authors. In [
The AILA 2021 Task 1 training data consists of 60 case documents. In each document the sentences are labelled by one of the 7 categories: 1. Facts: sentences that denote the chronology of events that led to filing the case 2. Ruling by Lower Court: the cases in the dataset were given a preliminary ruling by the lower courts (Tribunal, High Court etc.). These sentences correspond to the ruling/decision given by these lower courts. 3. Argument: sentences that denote the arguments of the contending parties 4. Statute: relevant statute cited 5. Precedent: relevant precedent cited 6. Ratio of the decision: sentences that denote the rationale/reasoning given by the Supreme
Court for the final judgement 7. Ruling by Present Court: sentences that denote the final decision given by the Supreme
Court for that case document
These documents were manually annotated by legal experts. In addition, we also included the rhetoric labels of the AILA 2021 Task 2 dataset and the AILA 2020 Task 2 dataset. The labels of these datasets were categorical variables so first we converted these variables to ordinal encoded variables. In total we had 12170 text-label pairs to train the model.
The task involves the classification of sentences from legal case documents into 7 rhetoric roles.
We use the simpletransformers library to import the BERT, RoBERTa, LaBSE and ALBERT
transformers which we use as classifiers. We then select the 3 best models and measure the
macro F1, precision, and recall values to find the best suited classifier for the task.
4.1. BERT
BERT[
The backbone of the ALBERT architecture is similar to BERT in that it uses a transformer encoder (Vaswani et al., 2017) with GELU nonlinearities (Hendrycks & Gimpel, 2016). The three main diferences are: • Splitting the embedding matrix into two smaller matrices using parameterized factorization embeddings • Cross layer parameter sharing • Inter sentence Coherence Loss
RoBERTa stands for Robustly Optimized BERT Pre-training Approach. optimize the training
of BERT architecture in order to take lesser time during pre-training. It has almost similar
architecture as compare to BERT, but in order to improve the results on BERT architecture, the
authors made some simple design changes in its architecture and training procedure. These
changes are:
• Removing the Next Sentence Prediction Objective
• Training with bigger batch sizes and longer sequences
• Dynamically changing the masking pattern
4.4. LaBSE
LaBSE [
We use these models as Multiclass Classification Models with the default parameters given by simple transformers.
The BERT, ALBERT, RoBERTa and LaBSE classifiers are compared using their mcc scores and evaluation loss on an evaluation data set. The results are shown in Table 1. The 3 best runs are chosen for predicting the rhetoric values for the given test data. The Precision, Recall and Macro F1-Score are calculated class wise and shown in Table 2. The overall score is then calculated and shown in Table 3. From Table 2, we can see that LaBSE has the best precision scores for Classifier
Epochs
Evaluation Loss Classifier
Metric
Argument
Facts
Precedent
BERT ALBERT RoBERTa
LaBSE 5 5 5 5 precedent and ratio of decision and has the best scores for predicting statutes. The LaBSE model does well in classifying statutes but poorly in classifying arguments whereas BERT was the best at classifying Arguments and the worst at Statutes. RoBERTa performs the best at ratio of decision and ruling by lower court classes. It also has highest precision with Facts. BERT performs consistently in most class predictions and has the best overall scores as shown in Table 3. All the classifiers struggle in the task of classifying decisions by lower courts with the highest F-Score being only 0.1346.
The task given was to semantically label the sentences in a legal document into 7 rhetoric roles.
Previously, the BERT and RoBERTa transformers, used as classifiers, produced the best results.
In this paper we use the simpletransformers library and import the classifiers BERT, ALBERT,
RoBERTa and LaBSE. First, we compare the runs of all 4 models and the 3 best performing
models were chosen to predict the test data set. In the test data set the BERT classifier performed
the best with RoBERTa being a close second. LaBSE outperformed the other two in predicting
statutes but performed significantly worse in classifying arguments and Ruling by Current
Court. The proposed method can be further improved by trying DynaBERT and ConvBERT
transformers [
We would like thank the Department Of Science and Technology (DST)-SERB funding scheme and HPC laboratory for providing the resources and space for our research. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. [14] V. Parikh, U. Bhattacharya, P. Mehta, A. Bandyopadhyay, P. Bhattacharya, K. Ghosh, S. Ghosh, A. Pal, A. Bhattacharya., P. Majumder, Overview of the third shared task on Artificial Intelligence for Legal Assistance at Fire 2021. In FIRE (Working Notes) - Forum for Information Retrieval Evaluation, India, December 13-17, 2021.
V. Parikh, U. Bhattacharya, P. Mehta, A. Bandyopadhyay, P. Bhattacharya, K. Ghosh, S. Ghosh, A. Pal, A. Bhattacharya., P. Majumder, FIRE 2021 AILA track: Artificial intelligence for legal assistance. In Proc. of FIRE 2021 - 13th Forum for Information Retrieval Evaluation, India, December 13-17, 2021. [15] P. Bhattacharya, P. Mehta, K. Ghosh, S. Ghosh, A. Pal, A. Bhattacharya., P. Majumder, Overview of the Fire 2020 AILA track: Artificial Intelligence for Legal Assistance. In FIRE (Working Notes) - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020.
P. Bhattacharya, S. Paul, K. Ghosh, S. Ghosh, A. Wyner, Identification of Rhetorical Roles of Sentences in Indian Legal Judgments. In Proc. of JURIX 2019 - International Conference on Legal Knowledge and Information Systems, 2019. [16] V. Parikh, V. Mathur, P. Mehta, N. Mittal, P. Majumder, LawSum: A weakly supervised approach for Indian Legal Document Summarization. arXiv preprint arXiv:2110.01188v3.