In the present attempt we build a Roberta based model for shared task 2 (Rhetorical Role Labelling for Legal Judgements) in AILA 20. We use Roberta model to get the text embedding. Output of the Roberta is passed through the bidirectional LSTM of 256 units and 128 units, after that passed that output through a dense layer and global_max_pooling_1D layer and finally a softmax layer of 7 activation unit. We use batch size of 16 and max length of 120. We submit 3 runs where 2nd, 3rd and 1st run submissions were scored (Macro-F-score) 0.468, 0.457 and 0.452 respectively. These three systems ranked 1st, 2nd and 4th respectively.
There are some previous work already done on this area. Saravanan et.al. [
2020 Copyright for this paper by its authors.
Legal case documents follow a common thematic structure with implicit sections like Facts of the Case, Issues being discussed, Arguments given by the parties, etc. These sections are popularly termed as "rhetorical roles".
So here we have to classify each sentence of a document to one of the below mentioned seven classes. Here we briefly explain what those seven classes actually mean for better understanding. 1. Facts: Sentences that denote the chronology of events that led to filing the case. 2. Ruling by Lower Court: Since we will be providing Indian Supreme Court cases, these cases were given a preliminary ruling by the lower courts (Tribunal, High Court etc.). These sentences correspond to the ruling/decision given by these lower courts. 3. Argument: Sentences that denote the arguments of the contending parties. 4. Statute: Relevant statute cited. 5. Precedent: Relevant precedent cited. 6. Ratio of the decision: Sentences that denote the rationale/reasoning given by the Supreme
Court for the final judgement. 7. Ruling by Present Court: Sentences that denote the final decision given by the Supreme Court for that case document.
We are given 50 legal documents for training purpose and 10 legal documents for testing. Number of sentences present in these documents are different. Each sentence is labelled with one of the seven classes. These classes are not balanced. Below we give a snapshot of our dataset that is provided us for training.
Pre-processing:- We are given labelled dataset. Labels of these dataset are categorical variables so first we have to convert these variables to one hot encoded variables. Here we have done nominal encoding (one hot encoding) instead of ordinal encoding because here order does not matter. Model:- We have tried different types of models for this shared task. Mainly those models are Random forest, Universal Sentence Encoder, BERT, ROBERTA. Among these models ROBERTA gives the best result.
In Random forest based model we used around 500 trees which are of full depth with gini index. In Universal Sentence encoder based model we have used U.S.E. large version 5, first we get those embeddings and then passed through the softmax layer of 7 activation unit to get the desired output. In BERT based model we used bert_large_uncased. We passed the output of the bert through the 256 unit neural network with relu activation unit then apply the dropout of 0.4 and passed it through the 128 unit neural network with relu activation unit and again apply dropout of 0.4 and finally passed it through the softmax layer of 7 activation unit to get the desired output.
In Roberta based model to predict labels the output of Roberta is send to the bidirectional LSTM of 256 units and 128 units respectively, after that passed that output through a dense layer and global_max_pooling_1D layer and finally a softmax layer of 7 activation unit. We have used loss function as categorical crossentropy and Adam as optimizer. We use batch size of 16 and max length of 120. We submit 3 runs where 1st run submission runs for 13 epoch, 2nd run submission runs for 15 epoch and 3rd run submission runs for 19 epoch.
Here we are going to see performance of our four different models (Random forest, Universal Sentence Encoder, BERT, ROBERTA) upon validation dataset, which we taken from our training dataset. From the below mentioned table we can see that among these models ROBERTA based model performed best.
precisio n recall F score precisio n
0.103
From the below mentioned confusion matrix (of ROBERTA model) we can see that from our validation dataset (10 documents of training set) which classes are predicted correctly and which are predicted wrong. We can see that Argument class predicted correctly 91 times, Facts class predicted correctly 388 times, Precedent class predicted correctly 149 times but precedent predicted as Ratio of the decision 221 times that is why we are getting recall value of 0.34, Ratio of the decision is correctly predicted 279 times but wrongly predicted as Facts 129 times, Ruling by Lower court predicted correctly for just 8 times and this creates the most problem for our models. This Ruling by Lower court predicted as Facts 39 times and Ratio of the decision 24 times so its recall value decreases very much and become 0.10 only. Ruling by present court predicted correctly 25 times and Statute predicted correctly 67 times.
For evaluation of the model Macro F-score is considered instead of accuracy because here given classes are not balanced. Here task organizers first calculate Recall, Precision and F-score for each category of labels within each document. Then the score for each document in a run were computed by averaging the scores for all seven categories in that document. Finally, the overall scores for a run are computed by averaging the scores for each document.
ju_nlp_1 ju_nlp_2 ju_nlp_3
0.504 0.506 0.519
0.483 0.452 0.501 0.468 0.479 0.457
0.588 0.588 0.57
Our system ju_nlp_2 ranked 1 in this competition with Macro F-score of 0.468. We submit our Roberta model with 13 epochs (ju_nlp_1), 15 epochs (ju_nlp_2) and 19 epochs (ju_nlp_3) respectively.
From the above we can see that we try different types of models to get desired output but we face main problem to detect the class Ruling by the lower court because in our validation dataset (comprised of 10 documents) we can only predicted it 8 times correctly and precision and recall value of 0.444 and 0.10 respectively. In future we should try some other things to detect this particular class correctly. Performance of this model depends upon how the test data reflects the real world dataset.