Legal query understanding is a complex problem that involves two natural language processing (NLP) tasks that needs to be solved together: (i) identifying intent of the user and (ii) recognizing entities within the queries. The problem equates to decomposing a legal query into its individual components and deciphering the underlying diferences that can occur due to pragmatics. Identifying the desired intent and recognizing correct entities helps us return back relevant results to the user. Deep Neural Networks (DNNs) have recently achieved great success surpassing traditional statistical approaches. In this work, we experiment with several DNN architectures towards legal query intent classification and entity recognition. Deep Neural architectures like Recurrent Neural Networks (RNNs), Long Short Term Memory (LSTM), Convolutional Neural Networks (CNNs) and Gated Recurrent Units (GRU) were applied and compared against one another both individually and as combinations. The models were also compared against machine learning (ML) and rule-based approaches. In this paper, we describe a methodology that integrates posterior probabilities produced by the best DNN models and create a stacked framework for combining the diferent predictors to improve prediction accuracy and F-measure for legal intent classification and entity recognition.
Copyright © CIKM 2018 for the individual papers by the papers' authors. Copyright © CIKM 2018 for the volume as a collection by its editors. This volume and its papers are published under the Creative Commons License Attribution 4.0 International (CC BY 4.0). 1.
With US legal services market approaching $500 bn
in 2018 that is 2% of US GDP [
In practice, a user has to decide his choice of vertical search engine beforehand to satisfy his query intent. It would be convenient if a query intent identifier could be provided in a general search engine that could precisely predict whether a query should trigger a vertical search in a certain domain. Moreover, since a user’s query may implicitly express more than one intent, it would be very helpful if a general search engine could detect all query intents, distribute it to appropriate vertical search engines and efectively organize the results from the diferent vertical search engines to satisfy a user’s need. Consequently, understanding a query intent is crucial for providing better search results and thus improving the overall satisfaction of the user.
Understanding intent and identifying legal entities from a user’s query can help legal search automatically route the query to corresponding vertical search engines and obtain relevant contents, thus, greatly improving user satisfaction meaning better assistance to law researches to support legal arguments and decisions. Law researchers have to cope with a tremendous load of legal content since sources of law can originate from diversified sources like judicial branch, legislative branch (statutes), legal reference books, journals, news etc. This makes legal search and understanding queries in legal search such an important aspect to retrieve relevant support documents for supporting one’s argument.
Legal search is hard as it demands writing complex queries to retrieve desired content from information retrieval (IR) systems. Classifying legal queries and identifying domain specific legal entities from queries is even harder. For e.g., in the query: “who is supreme court magistrate John Roberts and abortion law?”, the word “magistrate” can be resolved to a <judge title> when observed along with the context phrase “supreme court”. Similarly, the phrase “abortion law” can be identified as a <practice area> when seen alongside a supporting context. However, since we also observe the interrogative phrase “who is”, we can safely assume that the intent of this query is <judge> or <person> search.
Similar to google queries, legal queries can be classified into one of three categories. (i) Do: The users wants to do something, like buy a product or service. E.g., Buy Law Books, Research Guides, Police/Personal Reports, Real Estate Property Records etc. (ii) Know: An informational query, where the user wants to learn about a subject. E.g., law, statute, doctrine etc. Very often single word queries are classified at least partially as “Know” queries. (iii) Go: Also known as a navigational query, the user wants to go to a specific site scoped to a particular legal entity. E.g., a query where a user wants to know analytics of a specific legal entity like a judge or an expert witness. Our research in this work focuses only on “Go” type of queries, e.g., an user wanting to understand a particular judge profile with the query “ judge John D. Roberts”.
The process of finding named entities in a text and classifying them to a semantic type is called named entity recognition (NER). Legal NER is nearly always used in conjunction with intent classification systems. Given a query, the goal of NER system described in this paper is two-fold: (i) segment the input into semantic chunks, and (ii) classify each chunk into a predefined set of semantic classes. For e.g., given a query “judge john d roberts”, the desired output would be: “judge” = <OTHER> and “john d roberts” = <JUDGE>. Here the class <JUDGE> represents a person or a judge entity and the class <OTHER> represents any non judge specific term.
The aim of this research is to improve the understanding of queries involved in legal research. In this paper, we explore three diferent approaches: (a) machine learning (ML) with feature engineering, (b) deep learning (DL) without any feature engineering and (c) ensemble of deep neural networks. We then perform quantitative evaluations in comparison to an already existing baseline model which is a rule-based classification system in production. Finally, we show from our experimental results that a deep ensemble model significantly outperforms other approaches for both intent classification and legal NER. 2.
Background and Related Work
DL systems have dramatically improved the state of
several domains like NLP, computer vision, image
processing etc. Various deep architectures and learning
methods have been developed with distinct strengths
and weaknesses in recent years. Deep ensemble
learning is a learning paradigm where ensembles of several
neural networks show improved generalization
capabilities that outperform those of single networks. For
deep learning of multi-layer neural networks,
ensemble learning is still applicable [
Most ML approaches to text understanding consists
of tokenizing a string of characters into structures such
as words, phrases, sentences or paragraphs, and then
apply some statistical classification algorithm onto the
statistics of such structures [
Typical queries submitted to legal search engines
contain very short keyword phrases, which are
generally insuficient to fully describe a user’s information
need. Thus, it is a challenging problem to classify
millions of queries into some predefined categories. A
variety of related topical query classification problems
have been investigated in the past [
DNNs have revolutionized the field of NLP. RNNs and CNNs, the two main types of DNN architectures are widely explored to handle various NLP tasks. CNN is supposed to be good at extracting positional invariant features and RNN at modeling units in sequence. The state-of-the-art on many NLP tasks often switches due to the battle of CNNs and RNNs. While CNNs take advantage of local coherence in the input to cut down on the number of weights, RNNs are used to process sequential data (often with LSTM cells). RNNs are also good at representing very large implicit internal structures that are dificult even to think about.
In summary, the conventional wisdom is that RNNs should be used when the context is richer and there is more state information that needs to be captured. This proposition has been challenged by CNNs recently with the claim that finite state information of limited scope can be more eficiently handled by multiple convolution layers. We think both are true, and one should not go for RNNs just for the sake of it, instead more eficient deep CNNs should be tried in limited context situations. But for more complex implicit mappings where context and state information spans are much bigger, RNNs are the best and at this point almost the only tool.
There is not a lot of previous research work involving DL and legal domain for the problems of intent classification and NER. However, there are a handful of papers that talk about DNN approaches for nonlegal intent classification and general NER. Also recently, RNNs and CNNs have been applied on a variety of NLP tasks with various degree of success. Below, we talk about the evolution of various DNNs and their applications towards NLP tasks.
In [
In [
There are also many recent works that combine
RNNs with CNNs for diferent NLP tasks. For
example, in [
There is a handful of papers that delve into legal
applications using DNNs. In [
For NER, Huang et al. proposed a variety of LSTM
and LSTM variants like LSTM + CRF for NER and
achieved state-of-the-art results [
In this paper, we scope our research limited to judge queries meaning queries that are meant to be a search for judge profile. Being this the scope of the problem, the intent classification task needs to classify a given query as a judge query or not. Once the query is identified as a judge query, the NER system that follows, recognizes if the query contains any person entities. The entities are then routed to vertical searches that follow. The structure of the paper is as follows. Section 3. discusses the various experiments that was carried out to solve legal query understanding. That is followed by Section 4. that presents and analyses overall results. The paper ends with Section 5. which gives the conclusion and discussion on future work.
MODELS AND EXPERIMENTS The adoption of artificial Intelligence (AI) technology is undoubtedly transforming the practice of law. Many in the legal profession are aware that using AI can greatly reduce time and costs while increasing prediction measures. In the following subsections, we talk about data collection, augmentation and training of the diferent DNN models for legal tasks we experimented for: (i) intent classification and (ii) legal entity recognition. 3.1
For the purpose of our experiments, we created data sets comprising of diferent types of legal queries such as judge search, case law search, statutes/elements search, etc. For intent classification, since the classification task is binary, we labeled all judge queries to be positive and the rest as negative. For NER, we used three labels: (i) <OTHER> used to tag any tokens that are not part of a judge name (ii) <B-PER> denotes the beginning of a person (judge) name and (iii) <I-PER> denotes the inside of a name. Table 1 portrays the diferent query types by volume we used for our experiments. All of the collected data labeled by our subject matter experts (SMEs). All of the data discussed here are proprietary to LexisNexis. 3.2
The labeled data was good enough to get started but was highly imbalanced with respect to NER labels. In order to balance out the data set, we augmented the data by (i) expanding queries by pattern using judge names from our proprietary judge master database, (ii) oversampling the under represented patterns and (iii) under sampled the over exemplified patterns. The top 10 patterns by frequency are shown in table 3. These 10 patterns alone contributed to 78% of the labeled data. The balancing act was carried out by following two strategies: For oversampling, we pick a pattern and create synthetic data points (queries) by randomly substituting judge names/titles around the patterns to be fitted. For the process of under sampling, random data points are selected from buckets of pattern and removed. The sampling process is stopped when there are equal number of queries in all the buckets of patterns. Once the data is augmented and balanced, we split the data into train, dev and test sets. Table 2 shows the ratio of data split for both intent classification and NER. 3.3
Learning a high-dimensional dense representation for vocabulary terms, also known as a word embedding, Type Example Judge judge John D. Roberts Word Wheel Ohio Municipal Court,
Bellefontaine Query Log sexual harassment Statute statute /s limitations /s ac
tual /s fraud Elements Law contract defense uncon
scionability elements
Case Search Powers v. USAA
has recently attracted much attention in NLP and IR
tasks. The embedding vectors are typically learned
based on term proximity in a large corpus and is used
to accurately predict adjacent word(s) for a given word
or context [
Pattern B-PER I-PER I-PER
B-PER I-PER
O B-PER I-PER I-PER B-PER I-PER I-PER I-PER
O B-PER I-PER O B-PER I-PER I-PER I-PER I-PER
B-PER I-PER I-PER I-PER B-PER I-PER I-PER I-PER I-PER
O B-PER O O O B-PER I-PER I-PER I-PER O FP 57 137 1789 137 25 139 107
FN 209 141 20 256 647 142 160
F
For intent classification, we tried a few diferent ML classifiers firsthand before delving into the deep learning arena. The ML models were compared against a rule-based query recognition system highlighted that was already established as our baseline system to improve. Classifiers were selected from both the linear and non-linear category. The list of classifiers tried and their corresponding results (F1 score) against baseline are shown in table 4 above. Our main requirement while picking and implementing the classifiers was to minimize the overall number of false positives. This is because false positives with respect to intent classification can intercept queries that are meant to be routed to a diferent vertical search engine afecting the overall customer satisfaction towards the product. Amongst the linear classifiers, linear SVM seemed to perform slightly better than the logistic regression classifier. Amongst the non-linear category, the multilayer perceptron highlighted seemed to perform the best beating decision trees, adaboost and naive bayes approaches. 3.4.2
There are diferent ways one can address a judge in legal taxonomy such as “chief justice”, “associate justice”, “magistrate” etc. All of the diferent phrases used for addressing a judge are compiled into a bag of words representation. In addition to bag of words, all the ML classifiers use POS, gazetteer, word shape and orthographic features to represent semantic and linguistic meaning of a query. 3.4.3
The LSTM, first described in [
CNNs are built out of many layers of pattern recognizers stacked on top of each other. Convolutional is a way of saying that the machine looks at small parts of a query first rather than trying to account for the whole thing. Each successive layer combines information from these small parts to fill in the bigger picture and assemble complex patterns of meaning. After trying few variants of LSTM architectures, we started experimenting with CNNs for intent classification. The general neural architecture we used for CNN is show in figure 2. The major diference here compared to the LSTM models are: CNNs use two dense layers after the flatten layer whereas LSTMs use only one layer. The 1D convolutional layer uses 32 filters with a kernel size = 8 and the max pooling layer uses 2 strides with a pool size = 2. 3.4.5 As a next step, the best performing models from both the LSTM and CNN pools were picked and combined into a hybrid model for classification. We built two
Conv1D [32, 8, relu] MaxPooling1D [2, 2, valid] models for this experiment, LSTM + CNN and BiLSTM + CNN. The architectural diagram for BiLSTM + CNN model is shown in figure 3. 3.4.6
Ensemble learning is a ML paradigm where multiple learners are trained to solve the same problem. In contrast to ordinary ML approaches which try to learn one hypothesis from training data, ensemble methods try to construct a set of hypotheses and combine them to use. For intent classification, we use stacking which applies several models to original data. In stacking we don’t have just an empirical formula for our weight function, rather use a logistic regression model to estimate the input together with outputs of every model to estimate the weights or, in other words, to determine what models perform well and what badly given these input data.
To simplify the stacking procedure for ensemble LSTM Bi-LSTM
CNN
LSTM + CNN Bi-LSTM + CNN learning, we perform a linear combination of the original class posterior probabilities produced by the best DNN models at the word level (see table 8). A set of parameters in the form of full matrices are associated with the linear combination, which are learned using the training data consisting of the word-level posterior probabilities of the diferent models and its corresponding word-level target values (0 or 1). Figure 4 depicts the model architecture of the ensemble classiifer for the task of intent classification. 3.4.7
Table 5 below shows the overall training statistics for the diferent DNN models deployed for intent classification. The models were trained on AWS ml.p3.8xlarge instance with 4 NVIDIA Tesla V100 GPUs. Average time shown below is measured in minutes. 3.5 3.5.1
Baseline model for NER is shown in table 12 . The baseline model (highlighted in ) was previously established and it is the same rule-based system that was set as baseline for intent classification. The row highlighted in shows metrics from a conditional random Architecture Dropout
Hidden Units
Embedding LSTM I LSTM II LSTM III
LSTM IV Bi-LSTM I Bi-LSTM II
False False False True False True 50 50 50 50 100 100 200 100 200 200 200 200
Architecture Filters
Kernel Padding CNN I CNN II CNN III CNN IV CNN V 32 32 64 64 64 16 8 8 16 4 same same same same same LSTM III Bi-LSTM I
CNN II LSTM III + CNN II Bi-LSTM I + CNN II
Ensemble (Top 5) same same same same 100 100 100 100 100 100 100
FP 25 52 23 54 19 38 FN 13 13 11 15 9
F
FP 14 26 17 24 11 26 FN 18 18 18 15 14
FN 42 27 25 21 22 23 F-score ifeld (CRF) probabilistic model that was previously implemented. This model was eventually discarded since it was outperformed by most of DNN models. 3.5.2
For NER, we started with RNNs. The neural architecture for RNN is shown in figure 5 above. For the embedding layer, we used pre-trained word2vec embeddings of dimensions (577,149 x 200). For the RNN layer, 200 hidden units were used. The time distributed output layer has 3 units and uses a softmax function since we have 3 classes in total (<OTHER>, <B-PER> & <I-PER>. A dropout value of 0.2 was also used for the configuration.
LSTMs in general circumvent the vanishing gradient problem faced by RNNs. For NER, we used both uni and bi-directional LSTMs. The architecture is similar to shown in figure 5 except the RNN layer is replaced 3.5.4
For NER, we also experimented with CNN. The neural architecture for CNN is shown in figure 6. The Conv1D layer consists of 128 filters with kernel size set to 5. In contrary to the CNN used for intent classification, this architecture does not use a max pooling layer. 3.5.5
More recently, gated recurrent units have been
proposed [
Conv1D [128, 5, relu] To create an ensemble for NER, the DNN models are ranked by their F1 score. The top 5 best models are then picked and stacked into an ensemble. Ensemble of top 10 models was also experimented and discarded since it underperformed compared to the ensemble of top 5. Figure 8 shows the architecture of the chosen ensemble model. We utilize standard measures to evaluate the performance of our classifiers, i.e., precision, recall and F1measure. Precision (P) is the proportion of actual positive class members returned by our method among all predicted positive class members returned by our method. Recall (R) is the proportion of predicted positive members among all actual positive class members in the data. F1 is the harmonic average of precision and recall which is defined as F1 = 2PR/(P+R). Empirical results for intent classification are shown in tables 6, 7 & 8. Results for NER are shown in tables 9, 10, 11 & 12. Based on evaluation metrics, we can clearly see that the ensemble models outperform all other DNN models in both tasks, intent classification as well as NER by a good margin. In case of intent classification, the ensemble model (top 5) highlighted in table 8 has the lowest count of false positives and false negatives on both the dev and test data sets. It also has the highest F1 score value = 99.91% beating the baseline rule-based system by a margin of 1.5%. In case of NER, the ensemble model (top 5) highlighted in table 12 outperforms all other DNN models and beats the baseline model by a margin of 6%.
Conclusion and Future Work Our results show an ensemble model of stacking different DNNs of varying architectures outperforms individual performances of DNNs for the tasks of legal intent classification and entity recognition. RNNs, LSTMs, GRUs and even CNNs, all compress the necessary information of a source query into a fixed-length vector. This makes it dificult for the DNNs to cope with long queries, especially those that are longer than the queries in the training corpus. In future, we plan to use attention within queries. Attention is the idea of freeing a DNN architecture from the fixed-length internal representation. The DNN models we trained are at the word level, in future we plan to expand the size of the training data and try DNN models at the character level. Moreover, since the diference in performance between the DNN models were rather small, we plan to run tests of statistical significance and error analysis to capture performance by patterns. Lastly, we also plan to look into the impact on our models with respect to data and covariance shifts.
This research was supported by LexisNexis, Raleigh Technology Center, USA.