In this paper, we propose a legal text classification model based on text statistical features and deep semantic features, which is mainly used to solve the legal text classification task of FIRE2020(Forum for Information Retrieval Evaluation)@AILA. We use the TF-IDF feature of text as the statistical feature of the text, the [CLS] token information output by the BERT model as the deep semantic feature of the text, and the combination of the two as the joint feature of the text. Joint features are used to train classifiers such as Logistic Regression and SVM. Compared with the deep learning method and TF-IDF based method, the method using joint features has a greater performance improvement. The F1 score of this method on the test set reaches 0.457, which is the second in all the submitted teams.
The overall process of our method is shown in Figu1r.eAs shown in Figure1, our task is
regarded as a multi-classification task. The legal text is converted into statistical features and
semantic features to train a multi-classifier. Among them, the tf-idf vector of the text is used as
the statistical feature, and the CLS vector at the output of the BE2R]Ti[s used as the semantic
feature of the text. The multi-classification model uses logistic regressi3o]n,[support vector
machine[
The data set provided in this evaluation is divided into training data and test data. The training data set contains 50 files, 9380 pieces of training data, and the test data contain 10 files and 1905 pieces of test data. The training data is classified into seven categories. The proportion of each label is shown in Tabl1e.
1. Facts: sentences that denote the chronology of events that led to filing the case 2. Ruling by Lower Court: Ruling by Lower Court: since we will be providing Indian Supreme Court cases, these cases were given a preliminary ruling by the lower courts.
These sentences correspond to the ruling/decision given by these lower courts. 3. Argument: sentences that denote the arguments of the contending parties 4. Statute: relevant statute cited 5. Precedent: relevant precedent cited 6. The ratio of the decision: sentences that denote the rationale/reasoning given by the
Supreme Court for the final judgment 7. Ruling by Present Court: sentences that denote the final decision given by the Supreme
Court for that case document
To select the optimal experimental parameters, we divide the training data into a 7:3 ratio: training set (6566 items) and development set (2814 items), and select the final model hyperparameters based on the experimental results on the development set.
Our method is divided into three steps: use the training data to fine-tune the BERT based on the classification task. Use the BERT source code to extract the output results of the last layer of the model. Only the feature vector in the [CLS] token in the model is used as the deep semantic feature of the sentence, so the deep semantic feature of each sentence is a 768-dimensional vector. Next, use the scikit-lear1ntool to extract the TF-IDF features of the text as the text statistical features. Each input sentence can get the 5814-dimensional tf-idf features. Combine deep semantic features and text statistical features as the final 6582-dimensional joint text feature. Finally, the joint text features are input into a logistic regression, SVM, AdaBoost classification models for final training and prediction.
The parameters of the final model are selected according to the performance of each classifier on the development set. The fine-tuning parameters of the BERT model are selected according to the classification accuracy of the BERT on the development set. Since the maximum sentence length of 128 can cover 99.5% of the sentence length, because seq_length is directly set to 128, no parameter adjustment is required. The parameter settings and classification efects of the BERT model are shown in the following Tab2le(the main evaluation metrics is the Accuracy value).
Finally, select the 10th group of experimental parameters as the final BERT fine-tuning parameters, seq_length=128, Batch_size=32, Learning_rate=2e-5, and epoch=2. The main adjustment parameter of logistic regression and SVM is the C value, which is selected according to the F1-score of the two classifiers on the development set. Since both logistic regression and SVM need to set parameter C, because the two tuning experiments are combined and compared, the experimental feature is joint features, and the evaluation indicators are Accuracy and F-score.As shown in Figur2e.The parameter adjusted by Adaboost is the number of weak classifiers in the model n_estimators.As shown in Figur3e.
The final experimental parameter settings are shown in Tab3l,elogistic regression, SVM classifier only adjusts the value of parameter C, the parameter adjusted by Adaboost is n_estimators
Our final experimental results on the test set using joint features are shown in Ta4bl(seorted by F-score).In the experimental comparison between the Tf-Idf feature and joint feature, the evaluation index is the F-score value. The experimental result is the F-score on the development set.The experimental results are shown in Figu4r.e
Comparison of the experimental efects of various classifiers and various feature combinations under diferent parameters (take the best experimental results) and the improvement is calculated according to the corresponding model’s tf-idf feature F-score experimental result.As shown in Table5.
In this evaluation, we used joint features combined with logistic regression support vector machines and Adaboost to solve the multi-classification problem in legal texts. We use the training data to fine-tune the BERT model. After fine-tuning, we extract the [CLS] information output by the model as the deep semantic information of the sentence, combining the tf-idf feature of the text as the joint feature of the text. Use logistic regression, support vector machine, and Adaboost classifier for classification and prediction. On the development set, the classification accuracy of the BERT model is 0.63, the optimal accuracy result of the logistic regression classifier using only tf-idf features is 0.59, and the optimal classification accuracy of the joint features is 0.73, which is improved compared to the BERT model 15.9%, compared with the tf-idf feature increased by 23.7%, the performance improvement is obvious. This work is supported by the National Social Science Fund of China (No.18BYY125).