A law practitioner has to go through a lot of long legal case proceedings. To understand the motivation behind the actions of diferent parties/individuals in a legal case, it is essential that the parts of the document that express an intent corresponding to the case be clearly understood. In this paper, we introduce a dataset of 93 legal documents, belonging to the case categories of either Murder, Land Dispute, Robbery, or Corruption, where phrases expressing intent same as the category of the document are annotated. Also, we annotate fine-grained intents for each such phrase to enable a deeper understanding of the case for a reader. Finally, we analyze the performance of several transformer-based models in automating the process of extracting intent phrases (both at a coarse and a fine-grained level), and classifying a document into one of the possible 4 categories, and observe that, our dataset is challenging, especially in the case of fine-grained intent classification.
Category Corruption Land Dispute Murder Robbery
No. of documents 17 25 30 21
etc.), and only include the words expressing cor- Tesla P100 GPUs with 16 GB RAM to perform all the
responding intent. The resulting spans are the experiments.
intent phrases. Inter-annotator agreement
(Cohen ) is 0.79. 3.1. Document Classification
3. Sub-intent annotation: 1 annotator who is
aware of legal terminology, is asked to go through Recent advancements show that, Transformer [
Then, the intent phrases are divided amongst Model Name Accuracy these annotators, and the sub-intent of each intent phrase is annotated thereafter.
Table 1 shows the statistics of our dataset, describing the number of documents, average length of documents and intent phrases, and average sentiment score for each of the 4 intent categories. The documents on Corruption and Land Dispute are roughly longer than those on Table 2 Murder and Robbery. Table 1 also shows average senti- Results of Transformer Models ment scores across annotated intent phrases (calculated using sentifish 2 Python Package) for each of the four We then implemented diferent models mentioned in categories. The sentiment scores of the categories fol- Table 2, for learning contextual representations of the low the following order - Land Dispute > Corruption > documents whose outputs were then fed to a softmax Robbery > Murder, which follows common intuition. layer to get the final predicted class of the document.
Fig. 1 shows the top 200 most frequent words (exclud- Along with this, we also implemented a variant of
LEGALing stopwords) occurring in the intent phrases for each of BERT [
This section is organized to describe the use of
transformers [
LEGAL-RoBERTa The training method is same as that of BERT. Owing to the novel attention mechanism used in DeBERTa, it outperforms the other models in terms of both Accuracy and Macro F1-score.
LEGAL-BERT on the other hand is pre-trained and further fine-tuned on legal-domain specific corpora, which in turn lead to its state-of-the-art performance on various legal domain specific tasks. In our case, leveraging LEGAL-BERT outperforms other models since the contextual representation is more inclined towards legal matters.
All of the transformer models were implemented
using sliding window attention [
We report the Accuracy score and Macro average score
for each of the model so as to get an intuition on how
the state of art transformer-based architectures perform
on document classification in the legal domain.
3.2. JointBERT
We implemented BERT for joint intent classification and
slot filling [
The models were trained over 10 epochs with a batch size of 16, at a learning rate of 2e-5. At each epoch checkpoint, the model was saved and the model with the highest validation accuracy was picked to evaluate on the test set. As can be seen from Table 3, BERT proved to be the best model with an Intent Accuracy as well as Intent Macro F1-score of 0.9.
Table 4 gives the evaluation metric scores for each intent separately and the analysis provides evidence that the transformer-based models perform poorly on Corruption intent due to the number of ocuments in that
0.90 0.90 0.88 category being the lowest, whereas they perform significantly better on other intents.
Score for fine-grained intent classification for the best performing model among the three models, i.e., JointBERT with a BERT Backbone. The labels are presented in the form of _ , where is an intent (e.g. Robbery), and is a fine-grained intent/sub-intent (e.g. action). We observe that, even though the number of training samples per fine-grained class is quite low, performance on the test set is quite good - The F1-Score for all classes is above 0.4, and except for two classes, it is above the halfway mark of 0.5.
Note that we have not reported the slot classification results for the fine-grained intents. This is because the number of labels becomes almost twice in this case as compared to intent classification (due to the presence of both B and I tags corresponding to each fine-grained intent, and an O class additionally, as we consider BIO tags for annotation). Hence, the number of samples per class is insuficient to learn a good slot classifier. 4. Discussion We observe that, although transformer-based models are performing well in the case of document classification
Table 6 provides the classification accuracy and Intent and coarse-grained intent classification, there is a need Macro F1-score on fine grained Intent Classification task. for better performance in the fine-grained intent classifiAs the intent becomes more specific, the scores drop sig- cation case. Hence, we argue that our dataset could be a nificantly, showing that the models are unable to capture crucial starting point for research on fine-grained intent the in-depth context of the intent phrases. However, mo- classification in the legal domain. dle with the BERT backbone still performs the best. This can be attributed to the fact, that BERT has the high- 5. Conclusion est number of parameters ( 110 million) as compared to ALBERT ( 31 million), and DistilBERT ( 50 million).
0.46 0.48
0.40 0.47 This paper presents a new dataset for coarse and finegrained annotation, as well as, shows a proof-of-concept as to how document as well as intent classification can be automated with reasonably good results. We use diferent transformer-based models for document classification, and observe that DeBERTa performs the best. We try transformer-based models such as BERT, ALBERT and DistilBERT as the backbones of a joint intent and slot classification neural network, and observe that, BERT performs the best among all the three, both in coarse as well as fine-grained intent classification. However, our dataset is challenging, as there is a lot of scope of improvement in the results, especially in fine-grained intent classification. Hence, our dataset could serve as a crucial benchmark for fine-grained intent classification in the legal domain.