Automatic summarization of lengthy legal documents can provide great help to the involved legal practitioners, as well as all the other end users. In this work, an extractive summarization approach has been developed which represents legal document sentences in terms of domain specific pre-trained embeddings and performs subsequent multilayer perceptron based classification to find their summary worthiness. With this approach, we have participated in the summarization related shared tasks of AILA 2021 (Task 2(a) and 2(b)). The results on the test dataset for Task 2 show that our proposed approach is able to outperform most of the other competitors, achieving 2nd position in Task 2(a) and best ROUGE-F1 scores across all the ROUGE metrics for Task 2(b). While the proposed approach has produced impressive results for Task 2, the same approach could not do well for the rhetorical labeling task (Task 1) as per the results provided by the organizers on the test dataset.
Whereas, in extractive summarization, the main idea is to extract summary worthy sentences
from the input document itself to form a summary. Several research works have attempted to
summarize such lengthy legal documents [
Following the introduction, the organization of the rest of the paper is as follows: Section 2 presents a detailed methodology for the automatic legal document summarization task. Results and analysis are given in Section 3. Finally, Section 4 concludes this paper with a summarization of the findings along with potential future research directions.
The organizers of the AILA track have provided the training dataset for Task 1 and Task 2. For
Task 1, they provide 60 documents, where each sentence of a document is labeled with one of
the seven rhetorical labels (Facts, Ruling by Lower Court, Argument, Statute, Precedent, Ratio
of the decision, Ruling by Present Court) [12, 13]. For Task 2, the organizers have provided 500
document-summary pairs of judgments by the Supreme Court of India [14]. The organizers have
provided the pre-processed and sentence tokenized versions for both judgments and summaries,
along with summary-worthiness and rhetorical roles labels for each sentence in the documents.
A description of the AILA track FIRE 2021 is given in [
In order to perform the summarization task, we have randomly split the training dataset into five folds consisting of 400 training samples and 100 validation samples. Now, for each fold, we have performed the steps as shown in Fig. 1(a). All the sentences are firstly converted into contextual embeddings using a pre-trained Legal-Bert model [16]. In this way, a 768-dimensional vector is obtained since Legal-Bert has a fixed hidden size of 768 dimensions. These sentence embeddings are then fed through the MLP model as shown in Fig. 1(b). For this summary-worthiness classification problem, it is to be noted here that the last dense layer consists of single node with sigmoid activation function and binary cross-entropy loss.
Specifically for Task 2(a), firstly, contextual embeddings of every sentence in a document are found using the Legal-Bert pre-trained model. Then these embeddings are fed through the MLP 5 training folds Training documents (400 train, 100 val.)
Legal Bert embeddings
Train MLP model
Trained model Testing documents Dropout(0.4) Dense(128) Dropout(0.4) Dense(32) Dropout(0.4)
Dense(1) (b) MLP model model. Since it is a binary classification task, we have considered the dense layer of 1 node as the last layer with sigmoid activation and binary cross-entropy. This way, training is done for all five models (for five folds). We predict the summary-worthiness probabilities of each sentence of a test document using all the five models and take an average of all five probabilities to get an overall probability measure. We assign label ‘1’ to the sentence if the predicted average probability is greater than or equal to the threshold of 0.4, otherwise the ‘0’ label is assigned. This is the exact approach utilized for Run-1 submission of our team (nits_legal).
For Run-2 submission, instead of using a simple MLP model, we use a multi-task learning based MLP model. In this approach, we have used both rhetorical and summary-worthy labels and fed the training dataset into a multi-task learning MLP model. We have chosen to perform a multi-task learning based approach hoping that learning rhetorical labeling might be helpful for appropriately predicting the summary-specific relevance labels of sentences. The exact MLP model utilized for this is depicted in Fig. 2. The training of such a model again results in 5 diferent versions for five diferent folds, which enables averaging based model ensembling. At inference time, we predict in the same manner as that of Run-1 using all the individual multi-task learning models.
For Run-3 submission, we take the average of all the individual summary-worthiness probabilities resulting from each individual trained model of Run-1 and Run-2. If the average probability value is found to be greater than or equal to 0.4, we assign label ‘1’ to that sentence, otherwise a ‘0’ label is assigned.
Participating in the summarization task, we have performed a sentence classification based extractive summarization of legal judgement documents. We have made use of those models that have been saved for Task 2(a). We used those models corresponding to each Run during the testing phase and got the probabilities for each sentence of a document. We took average probabilities directly as scores for each sentence and ranked the sentences in decreasing order according to these probability scores. Sentences are then picked up according to desired ) 8 6 7 ( e s n e D .t() 4 0 u o p o r D ) 8 2 1 (s e n e D .t() 4 0 u o p o r D summary length (in number of words) as given by the organizers. For Task 2(b), we have submitted three runs of our proposed approaches each of which directly utilizes the individual models saved during the Task 2(a) submissions for Run-1, Run-2 and Run-3.
3.1. Setup For training an MLP model, we have chosen a batch size of 32 and adam optimizer with a learning rate of 0.001. We run the MLP model for 500 epochs with an early-stopping of 100 epochs. The classification task is evaluated using the standard metrics such as Precision, Recall, and F-score. In contrast, the summarization task (task 2(b)) is evaluated with the help of ROUGE metrics, which is a prevalent metric for evaluating automatically generated summaries. All the experiments have been performed with the help of a Linux based machine with RTX-2070 (8 GB GPU).
The results on the test dataset for Task 2 has been shown in Tables 1,2,3 and 4. Our team nits_legal, has ranked second for the Task 2(a), whereas the Run id 1 for Task 2(b) has achieved the highest scores in terms of ROUGE-F1 scores for all variants of ROUGE as shown in Table 2. In terms of ROUGE-recall, our team has achieved the highest scores for ROUGE-3 and ROUGE-4 metrics as shown in Table 3. Whereas, for ROUGE-precision, our team has achieved the second-highest scores for all variants of ROUGE metrics except ROUGE-1 metric as shown in Table 4. Please note that the best scores for each measure is in bold in the tables.
One of the key observations drawn from the results on the test dataset in the case of Task 2(b), is that there is not much diference between the performances of our proposed approaches across all the ROUGE-recall, precision, and F1-scores. All three metrics are very well-balanced, which demonstrates that our proposed approach is able to produce very precise summaries, while having decent recall values. Usually, when the target summary lengths are not known, top k words are taken during the summary formation step, where k depends on the average summary lengths in the training set. However, in this sub-task, the desired summary lengths for each test document were already given by the organizers, and in spite of such constraints, our model is able to achieve very well-balanced summarization.
We tried to apply the same MLP based classification approach for Rhetorical role labeling (Task 1) multi-class classification problem also. However, our approach could not perform as eficiently for this task as it did for the summarization specific tasks, as shown in Table 5. For this task, apart from the straightforward MLP based submission, we also submitted a minority class oversampling based run (Run id 2) which did improve the classification performances slightly. Interestingly, the best performing team’s prediction accuracy scores for the minority class (Ruling by Lower Court) is 0 across all the metrics, which is much worse than our results for the minority class. However, even with this improvement, the results were not very encouraging.
Chandigard_concordia Such reduced performance may be attributed to the fact that we have considered the rhetorical labeling task as a sentence level multi-class classification problem and not as a sequential sentence classification problem at the document level. In our proposed approach, although each sentence is represented by its contextual embedding, it still lacks the information on how these sentences contribute towards the overall document. Further explorations can be performed where the rhetorical labeling task is considered both at the sentence as well as at the document levels, in a hierarchical fashion. Please note that the highest overall precision, recall and F1-score is in bold in the case of Table 5.
Legal document summarization task becomes very important when there is unstructured and lengthy documents such as Indian case judgements documents. In this paper, we describe our methodology for summarization of legal documents, as part of the AILA shared task in FIRE 2021. We have explored the application of the Legal-Bert model to find efective sentence embeddings and then fed these embeddings as input to an MLP model for the purpose of extractive summarization. This kind of MLP based classification approach is found to be very efective at generating extractive summaries of legal judgement documents with very impressive ROUGE scores. Our proposed approach is able to obtain the best summarization scores among all the participants for most of the ROUGE metrics under consideration in Task 2(b). Moreover, for the sentence summary-worthiness prediction task (Task 2(a)) also, our proposed approach was able to attain the second position among all the participants. We found that even though this approach is very efective at summarization of legal documents, it is not as eficient at the task of rhetorical role labeling (Task 1).
In order to further improve the performances for both of the tasks, hierarchical representation of the documents should be taken into consideration, along with the exploration of some of the recent neural architectures such as Graph Neural Networks (GNN). [9] A. Farzindar, G. Lapalme, Letsum, an automatic legal text summarizing system, Legal knowledge and information systems, JURIX (2004) 11–18. [10] C. Grover, B. Hachey, I. Hughson, C. Korycinski, Automatic summarisation of legal documents, in: Proceedings of the 9th international conference on Artificial intelligence and law, 2003, pp. 243–251. [11] D. Jain, M. D. Borah, A. Biswas, Summarization of legal documents: Where are we now and the way forward, Computer Science Review 40 (2021) 100388. [12] P. Bhattacharya, P. Mehta, K. Ghosh, S. Ghosh, A. Pal, A. Bhattacharya, P. Majumder, Overview of the fire 2020 aila track: Artificial intelligence for legal assistance, in: FIRE (working notes), 2020. [13] P. Bhattacharya, S. Paul, K. Ghosh, S. Ghosh, A. Wyner, Identification of rhetorical roles of sentences in indian legal judgments, in: Proc. International Conference on Legal Knowledge and Information Systems (JURIX), 2019. [14] V. Parikh, V. Mathur, P. Mehta, N. Mittal, P. Majumder, Lawsum: A weakly supervised approach for indian legal document summarization, arXiv preprint arXiv:2110.01188v3 (2021). [15] V. Parikh, U. Bhattacharya, P. Mehta, A. Bandyopadhyay, P. Bhattacharya, K. Ghosh, S. Ghosh, A. Pal, A. Bhattacharya, P. Majumder, Overview of the third shared task on artificial intelligence for legal assistance at fire 2021, in: FIRE (Working Notes), 2021. [16] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, I. Androutsopoulos, Legal-bert: The muppets straight out of law school, arXiv preprint arXiv:2010.02559 (2020).