One of the fundamental assumptions with any machine learning (ML) system is that training data comes from the same distribution as the real world data. However, in many real-world applications, this important assumption is often violated including legal research. A scenario where training and test samples follow different input distributions is known as covariate shift . This shift in data is often responsible for the deterioration in predictive performance of machine learning systems. The motivation of this research is to study the efect of covariate shift on deep learning systems used in legal research. In this paper, we propose a uniifed framework to detect covariate shift impacting AI systems and formulate a strategy to adapt to this shift on a periodic basis. To our knowledge, our work is the first to apply data shift detection and adaption techniques to deep learning systems involving high dimensional word embeddings. Through experiments and evaluations, we demonstrate that our framework can accurately detect data (covariate) shift on legal AI systems involving deep neural architectures.
CALR stands for Computer Assisted Legal Research and is a mode of legal research that uses electronic databases that comprises court documents, statutes, secondary materials etc. Professional lawyers and paralegals rely on CALR applications for the precise understanding of the law and to serve the client’s best interest. Search engines are a crucial component of legal research technology today, and its primary goal is to identify and retrieve information needed to support legal decision making. When a user types the query “most cited cases by judge john d roberts”, he strives to understand the most often cited cases by a judge and tries to anticipate the judge’s behavior. This plays a crucial role in uplifting his legal research experience. Consequently, understanding a query intent is essential for providing beter search results, thus improving customers’ overall satisfaction.
Understanding a query intent requires classifying legal queries
and identifying domain-specific legal entities, which is a complex
problem [
Example marbury v. madison, 1803 judge john roberts expert henry lee foreign corrupt practices act of 1977 ? seminal cases on murder burden of proof for hearsay statement what is assumed duty doctrine elements of child abuse statute of limitations for mail fraud
Table 1: Query Intents
Identifying query intent is a classification problem, and the process of recognizing domain-specific entities is known as named entity recognition (NER), which also belong to the classification family. In general, intent and entity recognition are two primary components of any natural language processing (NLP) system. Over the past decade, the field of NLP has heavily influenced the way legal search works, shifting discovery from pure keyword-based methodologies to a more context-oriented NLP techniques.
Figure 1 depicts the workflow of a typical CALR application. The browser is a tool by which users provide input; the application layer coordinates user interactions with a service layer that triggers a search. The NLP system complements the service layer for query understanding by leveraging a database layer. The database layer usually retrieves relevant information in the form of legal documents.
NLP systems are usually built using supervised approaches and is a type of learning that uses a function to map a given input to an output. It infers learning features from labeled data consisting of training examples. For example, given a query “what are the opinions by judge john doe in civil cases dealing with dog bites ?”, the Case Law
Docs Legal Environment
Machine Learning Development Life Cycle output label for intent classification is “ judge”. On the other hand, we need to construct two output labels “john doe” as <judge entity> and “civil cases” as <practice area> for the task of legal entity recognition.
Our NLP system mainly consists of two models: (i) a model for identifying legal query intent which we call as Query Intent Classifier (QIC), and (ii) a model for recognizing legal entities called as Legal Entity Recognition (LER). QIC and LER model development cycles follow a standard machine learning development life cycle as shown in Figure 2. These cycles usually require application data as a prerequisite. In our case, the data is derived from user logs as mentioned previously. NLP systems usually reside within a learning environment and learn from the data collected within this environment.
In general, any legal learning environment is comprised of users (lawyers and paralegals), continually changing legal corpus and reformulating legal queries. Most machine learning methods assume the learning environments to be static, which is not the case with real-world applications such as CALR, email spam filters, stock market prediction systems etc. Real world applications including legal systems are mostly dynamic in nature and often incur distribution changes to its underlying data. This phenomenon is known as data shift in the machine learning arena. These data shifts usually results in performance degradation of NLP systems deployed as real-world applications.
hTe evolving nature of legal environment demands continuous monitoring and adaption to data shifts, in order to alleviate the issue of performance degradation in NLP systems. Data shift has been receiving significant atention in recent years amongst the machine learning community. Dataset shift refers to the problem where training and real-world datasets follow diferent distributions. The section 2. contains formal definitions of these shifts. Since this problem may occur in many real-life scenarios, detecting and adapting to dataset shift becomes a vital research aspect in machine learning. This research aims to observe, detect and adapt covariate shift on deep learning models using high-dimensional word embeddings, derived from a corpus of legal queries. We demonstrate the usefulness of adapting covariate shift with incremental learning on deep neural models as a necessary step to ensure consistent quality amongst AI applications deployed for legal CALR. 2.
Dataset shift research on machine learning classification algorithms
is interesting and foreseen to become a more dificult problem to
solve in non-stationary environments. In section 2.1, we introduce
the dataset shift definitions concerning classification problems from
[
In this section, we explain the diferent classification and data shift types. In general, a classification problem is defined by:
A target (class) variable Y
A joint distribution P (Y ; X )
X ! Y problems are those where the class labels Y is predicted based on the values of covariates X . Inversely, Y ! X problems are the ones where the class label Y causally determines the values of covariates X . Thus, by analyzing the relationship between X and Y , we can define three diferent types of data shifts: 2.1.1 Covariate Shift : Covariate shift refers to changes in the distribution of input variable X = fx1; x2 xn g. Here x1; x2 xn are called the covariates and any distribution changes in one or more of these covariates is termed as covariate shift. 2.1.2 Prior Probability Shift : Distribution changes to the class variable Y is referred to as prior probability shift, and it appears only in Y ! X problems. 2.1.3 Concept Shift : Concept shift occurs when the relationship between input variable X and class variable Y changes. There other shifts in theory, but we are not defining them since they emerge sporadically. In general, dataset shift is a phenomenon that occurs when new data distribution leads to a change in the distribution of a single feature, a combination of features, or class boundaries.
Word Embedding
User Query Test Phase
Input Layer Embedding Layer
RNN Layer Output Layer
Shift Model Selection bias and non-stationary environments are the two primary reasons for data shift in general. Selection bias occurs when a training set does not exactly represent a real world test set. Lack of randomness in training sample selection, improper samples, and biased sampling rules often influence selection bias. On the other hand, non-stationary environments often must deal with dynamic nature. Hence, not handling dataset shift in real-world application creates an overfitted model on training samples, hence unreliable model predictions. 2.3
In this section, we introduce an overview of three well known covariate shift analysis methodologies. 2.3.1 Visualization: This methodology is the simplest, visualizing one covariate at a time. It requires humans spoting the diference in covariate distribution using histograms. 2.3.2 Statistical Distance: In this type of analysis, methods involving statistical metrics such as mean, variance, population stability index (PSI), Kullback-Leibler divergence, and KolmogorovSmirnov etc. are used for detecting shifts. 2.3.3 Uncertainty Quantification : This method fits a probabilistic model on the training data and every prediction on new data is associated with a confidence interval or uncertainty. Lower uncertainty on new real world data is considered no shift and higher uncertainty means a shift.
All the methodologies mentioned above have a common drawback of not being suitable for the analysis of high dimensional features. In our work, we strive to overcome this drawback by defining a shift detection algorithm to capture performance degradation in real world machine/deep learning systems, especially scoped towards legal data.
Data (covariate) shift is an area of machine learning that has been
gaining popularity in recent years. In this section, we will discuss
some of the very few existing works related to this sub-domain
of AI research. First, statistics based identification methods have
been widely adopted in several fields recently yielding good
results. For the area of data shift, in [
Most real-world deep learning applications need training and
the training phase usually face an internal covariate shift. In [
In [
Fm
R
training set. When adequate samples are available for the
training set, active learning is adapted. Active learning selects test
instances that dramatically influence the learning process and hopes
to reduce the uncertainty under covariate shift. Some earlier works
[
In this paper, we scope our research to handling covariate shift in word embeddings acquired from legal search queries. It also discusses an incremental learning approach for adapting covariate shift to legal AI systems. To our knowledge, this work is the first of its kind to apply data shift on word embeddings focussing on deep learning applications. It is also the first to apply it to legal domain space.
Deep learning (DL) systems learn representations of data with multiple levels of abstraction and are composed of several processing layers. These methods have dramatically improved the state-of-theart in NLP empowered by word embeddings. Learning a high dimensional dense representation for vocabulary terms, also known as a word embedding, has recently atracted much atention in NLP and information retrieval tasks. The embedding vectors are typically learned based on term proximity from a large corpus and are used to accurately predict adjacent word(s), given a word or context.
For the purpose of this study, we consider two NLP models which we had developed earlier: (i) a model for identifying legal query intent namely Query Intent Classifier (QIC), and (ii) a model for recognizing legal entities termed as Legal Entity Recognition (LER). Our DL models follow a similar architecture as described in Figure 3. They consist of four layers - input, embedding, recurrent neural network (RNN) and an output layer. The input layer receives a dense representation of the word vocabulary derived from legal queries contained in user logs. The vocabulary is a diverse mixture of legal query types (intent), e.g., judge queries, case search, legal definitions and others shown in Table 1.
For the embedding layer, we use pre-trained word embeddings
trained via a word2vec[
hTe Problem Legal data, in general, is both complex and diverse. User queries and word vocabularies extracted from these queries change over time. This in turn leads to changes in the underlying word embeddings which are usually the core components behind AI system(s). A word’s embeddings are vectors that represent some aspect of its meaning and are generally trained on large, unlabeled corpora (in our work - legal queries). Any change in word embeddings results in complications and inconsistencies within feature weights that are part of the embedding matrix. Furthermore this change makes it harder to accomplish a consistent prediction model whose behavior does not change frequently in production.
Our legal queries were both natural and un-natural, meaning, natural queries are mostly synthesized (augmented by us) and the un-natural queries are those derived from user logs. Users tend to type queries in more un-natural format. E.g., the query, “justice marshall abortion law 2017” is a very un-natural representation of language having multiple intents. Also, more structured queries like boolean queries are also un-natural in its representation.
Word embeddings are almost universally useful across a wide range of tasks, but there were many limitations to this method. Word embeddings are generally used for shallow language modeling tasks, so there is a limitation to what the word embeddings can capture. Unlike RNNs and other complex architectures, language models like word2vec have trouble capturing the meaning of combinations of words, negation, etc. On the other hand, instead of training a model to map a single vector for each word, RNNs learn to map a vector to each word based on the entire sentence/surrounding context.
Another key limitation is that word embedding models do not take context into account. For instance, the word “lynch” has different meanings. According to California Penal Code 405a, “Lynch” is defined as, “the taking by means of a riot of any person from the lawful custody of any police oficer.” It also refers to killing someone without legal authority, usually by hanging; and “lynch law” refers to the punishment of presumed crimes or ofenses, usually by death, without due process of law. The above example illustrates word embeddings built on non-stationary legal vocabulary are susceptible to data shift. Therefore, it is a necessity to develop strategies and techniques to overcome the issue. In the next section, we discuss the performance degradation of our DL models observed over 8 months of experimentation showing a covariate shift. 4.3
To evaluate our DL models, we chose F 1 score as a metric. In Table 2, m0 denotes the initial 0th month, and Fm0 denotes the corresponding F 1 score for models - QIC and LER for that month (m0). After development, the models were deployed to production in month m0. At m0, QIC’s Fm0 = 0:9344 and LER ’s Fm0 = 0:8733. hTese F 1 scores set the baseline for performance comparisons, and ∆ represents performance gain or loss correlated to these baselines.
hTe initial performance score observed during model development was good, significant performance degradation was observed after month m0, QIC and LER model performance scores and the observed shifts (delta values) for months m1 to m8 are shown in Table 2. Months m1 and m5 saw no significant performance degradation. Remaining months {m2,m3,m4,m6,m7,m8} are the ones with significant performance degradation. Month m6 witnessed a maximum QIC degradation where ∆ = 0:2302 and month m2 observed a maximum LER degradation whose ∆ = 0:2578.
Although, we achieved good baseline performance results, the degradation after deployment prompted us to research on the cause of degradation. Our analysis discovered that covariate shift in legal user queries consequently influences legal word embeddings causing distribution changes. These changes include observing vocabulary diferences such as new words or part of speech paterns. hTere is not enough research to identify word embedding distribution changes or covariate shift in legal (or any) environment. In this work, we propose a unique algorithm to detect covariate shift in the legal queries as explained in the upcoming section. 4.4
Previously, we talked about how our system sufers from covariate shift and Table 2 showed changes in F 1score clearly showing performance degradation. Our proposed algorithm detects covariate shift in legal user queries that impacts high dimensional word embeddings that derive from it. The degradation part was discussed earlier. This section first introduces the intuition behind the algorithm. Next, it discusses notations used for defining the algorithm, followed by explanation and results. 4.4.1 Intuition: The core intuition behind the proposed algorithm is to detect covariate shift by classifying “ new” real-world data (legal queries) as similar or diferent to the training data (“ old”). hTe algorithm starts with building a binary classifier over the combined dataset (proprietary to LexisNexis) of current training and monthly user queries, and then predicts a probability that a user query is a member of a training set. We assign membership to both training (old) and real-world (new or test) data with output labels - train and test to create input-output pairs for the shift classifier (binary). Inconsiderable training error (covariate shift) in new user queries and limited accuracy indicates that new real-world data and training data are similar. If significant word distributions shift in the real-world test queries, then the classifier correctly classiifes test queries from training queries, hence proving a distribution change in the covariates or word embeddings.
Symbol Xt Yt (X ; Y )t Xm Ym (X ; Y )m Xtm Ytm (X ; Y )tm M ∆ F ∆
Usage Variable denoting current training set of user queries Target variable denoting output labels for Xt Variable denoting input-output pairs for Xt and Yt Variable denoting current month’s user queries Target variable denoting output labels for Xm Variable denoting input-output pairs for Xm and Ym Variable denoting combined queries of Xt and Xm Target variable output labels for Xtm Variable denoting input-output pairs for Xtm and Ytm Binary classifier for the shift model F 1-score of M ∆ Mathews Correlation Coeficient of M ∆ 4.4.2 Notations: Table 3 displays the list of symbols used for defining the algorithm. In general, X as input variables and Y as output variables are the X and Y to the binary classifier M∆(shift model). Along with the standard F 1 score, an additional qualitative measure called Mathews correlation coeficient is also used for assessing M∆. For this work, we utilized a supplementary ∆ symbol for all shift model related symbols. Also sufixes t and m serve as current training and current month respectively.
Output: YES if covariate shift, NO otherwise 1 Assign target labels 0 to Yt and 1 to Ym 2 Combine inputs Xt and Xm along with their respective output labels Yt and Ym to create dataset (X ; Y )tm 3 Perform the classic train-test split on dataset (X ; Y )tm to create train set (X ; Y )t∆rain and test set (X ; Y )t∆est 4 Train a Word2Vec model using both Xt and Xm 5 Using word embeddings as features from Step 4, create a shift detection model M∆ trained on (X ; Y )t∆rain and tested it on (X ; Y )t∆est 6 Compute F 1 Score F ∆ and using the expected labels from
Step 1 and the predicted labels from Step 5.
7 If F ∆ > 0.7 and > 0.2 then return YES else return NO m1 m2 m3 m4 m5 m6 m7 m8
F ∆ 0.5 0.96 0.53 0.87 0.51 0.92 0.89 0.98 hTe architecture of our proposed shift model is shown in Figure 3 and it resembles the architecture of the previously discussed DL models (QIC and LER), (i) an input layer, (ii) an embedding layer in the form of pre-trained word2vec embedding, (iii) RNN layer consisting of LSTM units, and (iv) an output layer with sigmoid function that outputs a binary membership (0 or 1). Tokenized input queries are passed to the input layer along with its output labels, using which the shift model predicts membership (0 or 1) for the given query. The next step in the algorithm is to train a classifier M∆ on (X ; Y )t∆rain and test it on (X ; Y )t∆est . To determine covariate shift, we calculate standard F 1 test score F ∆ and shift score on the test set using expected labels assigned initially. If F ∆ > 0.7 and > 0.2, then algorithm returns “YES” as an indication of covariate shift and “ NO” otherwise. With trial and error, we arrive at the requirement that F ∆ should be 0.7 and also should exceed 0.2 to result in a covariate shift, and these values provide excellent coverage detecting the covariate shift for our application. 4.4.4 Results: Using our proposed algorithm, we conducted detection tests for 8 months as mentioned in Table 2, and Table 4 outlines the results of the tests. It comprises a shift score based on Mathews Correlation Coeficient, Fm∆ denoting shift model’s F 1 score on current training Xtm and the column “Is Covariate Shift” indicating “Yes” if covariate shift and “No” otherwise. Results from Table 4 shows the algorithm was able to detect covariate shift accurately for all months except month m3. Although the performance for month m3 has decreased, shift detection failed to detect this with a shift score =1.7 that did not meet our threshold. alarms and result in no efect on system’s performance, whereas false negatives result in system’s performance degradation.
R = NuTmotbaelrNoufmSbheiftsr DofetSehcitftsed X 100 (1) For our shift detection, we achieved a recall of 80% as show in Table 5. Thus, CSD acts as an alarm and it is the first process in the two-step procedure to learn under covariate shift. In the next section, we explain how we adapt to covariate shift through incremental learning. 4.5
hTis is the second stage of our proposed framework that facilitates adapting to covariate shift through incremental learning. Incremental learning is a method where the model’s knowledge base is updated continuously.
For incremental learning, at each iteration of CSD, Xt is updated on a monthly basis. Table 6 shows the cumulative Xt size of training data after each incremental learning. The baseline training set holds 1; 026; 883 queries at first month m0 and through incremental learning, the training set is accumulated to 2; 629; 006 at last month m8. We address covariate shift in legal queries by incremental re-training at each detection. To retrain our QIC and LER models, we used an expert system which is a Java-based rule engine. Subject Mater Experts (SMEs) produced the desired output labels Ym . The performance improvements after re-training is displayed in Table 7. It contains two F 1 scores, (i) an original F 1 score Fm from Table 2 before retraining and, (ii) a new latest F 1 score LMFm after retraining. Original performance degradation is ∆ from Table 2 and the result of incremental learning is measured by subtracting the latest F 1 score from the base F 1 score i.e., δ =FLm - Fm .
Apart from overcoming the performance degradation, incremental learning improved the F 1 score over the baseline. QIC has the highest improvement δ = +0:144 on m8 and LER has δ = 0:0191 on month m2. Our experimental results demonstrate the efectiveness of the proposed covariate shift-detection and incremental learning strategy. 4.6
For training our DL and shift models, we used AWS ml. p3:8x large instance with 4 NVIDIA Tesla V100 GPUs. Average time taken for all models is 94 minutes for 100 epochs, and training time for
Fm ∆
LER 0.8964 0.8790 0.8400
We utilize standard measures to evaluate the performance of our
P ), recall (R), and F 1-measure. Precision (P ) is the proportion of actual positive class members returned by our method among all predicted positive class members returned by our method. Recall (R) is the proportion of predicted positive members among all actual positive class members in the data. F 1= 2P R/(P +R) is the harmonic average of precision and recall. We also utilized Mathews correlation coeficient (
MCC) to compute the shift score and MCC is a correlation coeficient between actual and expected predictions. It varies between -1 and +1: -1 when actual and expected are entirely different, 1 when there is a perfect match and 0 when it is random.
was used as shift score for measuring the similarity between training and test set.
Learning strategies under covariate shift have been receiving significant research interest recently. In non-static environments such as legal, learning methods need to employ unique learning strategies and covariate shift monitoring systems to acquire a greater capability to generalize the learning. Our proposed framework in this work belongs to the category of incremental learning under covariate shift for legal AI systems, and its core component is a covariate shift detection algorithm which detects shift in our deep learning models - QIC and LER. The results demonstrate the benefit of building a monitor system for covariate shift detection and also its adaptation through incremental learning. For future work, we plan to extend our current research by employing beter strategies to reduce false negatives. Also, we are plan to apply a similar strategy to other data shifts such as prior probability and concept shifts.