Collections of documents filed in courts are potentially a rich source of information for citizens, attorneys, and courts, but courts typically lack the ability to interpret them automatically. This paper presents technical approaches to three applications of judicial document interpretation: detection of document filing errors; matching orders with the motions that they resolve; and predicting the outcome of routine cases. In empirical evaluations on filings from two representative large US District Courts, the highest accuracy in identifying filing errors was achieved by combining procedural context features with high information-gain lexical features; TF/IDF similarity was found to be an effective criterion for finding motions that correspond to orders; and induction over the texts of prior simple and routine decisions was found to produce a model capable of accurately predicting outcomes from case facts without any manually engineered features or factors.
The transition from paper to electronic filing in national, local, and administrative courts, which began in the late 1990s, has transformed how courts operate and how judges, court staff, attorneys, and the public create, submit, and access court filings. However, despite many advances in judicial access and administration brought about by electronic filing, courts are typically unable to interpret the contents of court filings automatically. Instead, court filings are interpreted only when they are read by an attorney, judge, or court staff member.
Machine interpretation of court filings promises a rich source of information for improving court administration and case management, access to justice, and analysis of the judiciary. The development of large-scale text analytics makes such interpretation increasingly feasible, as collections of court documents are, in effect, annotated by the metadata generated when they are submitted, by corrections when they are audited, or, for those documents that are motions or claims, by the decisions of judges or other decision makers.
However, there are numerous challenges to automating the interpretation of case filings. Courts often accept documents in the form of PDFs created from scans. Scanned PDFs require optical character recognition (OCR) for text extraction, but this process introduces many errors and does not preserve the document layout, which contains important information about the relationships among text segments in the document. Moreover, the language of court iflings is complex and specialized, and the function of a court filing depends not just on its text and format, but also on its procedural context. As a result, successful automation of court filings requires overcoming a combination of technical challenges.
This paper describes the nature of court dockets and databases, sets forth three classes of representative judicial document analysis tasks–docket error detection, order/motion matching, and decision prediction–proposes technical approaches to each of the tasks, and presents preliminary empirical evaluations of the effectiveness of each approach. 2
A court docket is a register of document-triggered litigation events, where a litigation event consists of either (1) a pleading, motion, or letter from a litigant, (2) an order, judgment, or other action by a judge, or (3) a record of an administrative action (such as notifying an attorney of a filing error) by a member of the court staff. Each docket event in a typical electronic case management system includes (1) metadata generated at the time of filing, including both case-specific data (e.g., case number, parties, judge) and event-specific data (e.g., the attorney submitting the document, the intended document type) and (2) a text document in PDF format. Each of the two court databases in which the experiments described below were performed contained filings for over 400,000 cases involving over 1,000,000 litigants, attorneys, and judges, over 10,000,000 docket entries, and more than 4,000,000 documents. 3
There are many kinds of docket errors, including defects in a submitted document (e.g., missing signature, sensitive information in an unsealed document, missing case caption) and mismatches between the content of a document and the context of the case (e.g., wrong parties, case number, or judge; mismatch between the document title and the document type asserted by the user). Some errors consist of violations of a particular court’s local rules and are therefore unique to that court. Other events, such as filing in a wrong case, constitute errors in any court. In either case, detection of defects at submission time could spare attorneys the embarrassment of submitting a defective document and the inconvenience and delays of refiling. For court staff, automated filing error detection could reduce the quality control (QC) auditing staff required for filing errors, a significant drain of resources in many courts. 3.1
In the court in which the first set of experiments were conducted, the QC staff review filings to detect a variety of docket errors, including the following four error types: • Event-type errors, i.e., specifying the wrong event type for a document, e.g., submitting a Motion for Summary Judgment as a Counterclaim. In experiments involving this court, there were 20 event types, such as complaint, transfer, notice, order, service, etc. • Main-vs-attachment errors, i.e., filing a document, such as an exhibit, that should be filed as an attachment to another document, as a main document or filing a document, such as a Memorandum in Support of a Motion for Summary Judgment, that should be lfied as a main document, as an attachment. • Show-cause order errors. In some courts, only judges are permitted to file show-cause orders; it is an error if an attorney does so. • Letter-motion errors. In some courts, certain routine motions can be filed as letters, but all other filings must have a formal caption. Recognizing these errors requires distinguishing letters from non-letters.
Event-type errors appear to be the most common docket errors in U.S. District courts.
Each of these filing errors can be detected by classifying a document with respect to the corresponding set of categories (event type, main vs. attachment, show-cause order vs. non-show-cause order, or letter vs. non-letter) and evaluating whether the category is consistent with the metadata generated in the docket system by the ifler’s selections. Event-type document classification is particularly challenging both because document types are both numerous and skewed, having a roughly power-law frequency distribution in the test set.
The first set of experiments attempted to identify each of the four docket errors above by classifying document text and determining whether there is a conflict between the apparent text category and the document’s metadata. Classification was performed with the lingpipe1 LMClassifier, which performs joint probability-based classification of token sequences into non-overlapping categories based on language models for each category and a multivariate distribution over categories.
3.1.1 Term Selection and Document Truncation. Court filings can be thought of as comprising four distinct sets of terms: • Procedural words, which describe the intended legal function of the document (e.g., “complaint,” “amended,” “counsel”) • “Stop-words," which are non-content common words, such as “of” and “the” • Words unique to the case, such as names, and words expressing the narrative events giving rise to the case; and • Substantive (as opposed to procedural) legal terms (e.g., “reasonable care,” “intent,” “battery”).
Terms in the first of these sets–procedural words–carry the most information about the type of the document. These words tend to be concentrated around the beginning of legal documents, often in the case caption, and at the end, where distinctive phrases like “so ordered” may occur. 1http://alias-i.com/lingpipe/
We hypothesized that only procedure terms are relevant to the
type of a document, so we explored approaches to filtering
nonprocedure terms. Elimination of irrelevant terms can not only speed
execution, but in some cases has been shown to increase accuracy
[
Three approaches to term selection were investigated: two ad hoc
and domain-specific; and one general and domain-independent. The
ifrst approach was to eliminate all terms except non-stopwords that
occur in the Federal Rules of Civil Procedure [
Figure 2 illustrates the reduction of full document text to just high information gain terms, which typifies the vocabulary-reduction process.
Several approaches to document truncation were explored as well. The first was to limit the text to the first l tokens of the document (i.e., excise the remainder of the document). If l is sufficiently large, this is equivalent to including the entire document. A second option is to include the last l tokens of the suffix as well as the prefix.
3.1.2 Evaluation of Alternative Term Reduction Approaches.
Two different information-gain thresholds were tested for each
classification type, intended to create one small set of very-high
information terms (ig_small) and a larger set created using a lower
threshold (ig_large). The thresholds and sizes of the large and small
high information-gain term sets are set forth in Table 1. The text of
each document was obtained by OCR using the open-source program
Tesseract [
Figure 3 shows a comparison of four vocabulary alternatives on the four text classification tasks described above. These tests measured mean f-measure in 8-fold cross validation using a 4-gram language mode, 50-token prefix length, and no suffix. In the baseline vocabulary set, normalize, non-ASCII characters, numbers, and punctuation are removed and tokens were lower-cased. The results show that classification accuracy using an unreduced vocabulary was significantly lower than the best reduced vocabulary performance for show-cause order detection and type classification. Term selection had little effect on accuracy for the letter and main vs. attachment detection tasks. No reduced-vocabulary set consistently outperformed the others. This indicates that restricted term sets derived through information gain perform roughly as well as those produced using domain-specific information, suggesting that the reduced vocabulary approach is appropriate for situations in which domain-specific term information is unavailable.
Summarizing over the tests, the the highest mean f-measure based on text classification alone and the particular combination of parameters that led to this accuracy for each classification task were as follows: (1) Event type: 0.743 (prefix=50, 4-gram ig_large vocabulary, 20 categories) (2) Main-vs-attachment: 0.871 (prefix=256, 6-gram, event vocabulary) (3) Show-cause order: 0.957 (prefix=50, 5-gram, ig_small vocabulary) (4) Letter-vs-non-letter: 0.889 (prefix=50, no 4-gram, ig_large vocabulary) 3.2
The accuracy of event-type detection (f-measure of roughly 0.743 under the best combinations of parameters) is sufcfiiently low that its utility for many auditing functions may be limited. An analysis of the classification errors produced by the event-type text classiifcation model indicated that a document’s event type depends not just on the text of the document but also on its procedural context. For example, motions and orders are sometimes extremely similar because judges grant a motion by adding and signing an order stamp to the motion. Since stamps and signatures are seldom accurately OCR’d, the motion and order may be indistinguishable by the text alone under these circumstances. However, orders can be issued only by a judge, and judges never file motions, so the two cases can be distinguished by knowing the filer. In addition, attachments have the same event type as the main document in CM/ECF. So, for example, a memorandum of law is ordinarily a main document, but in some courts a memorandum can be filed as an attachment, in which case its event type is the same as that of the main document to which it is attached.
Contextual information potentially relevant to a document’s type includes: whether it was filed as a main document or as an attachment; the filer (e.g., attorney, clerk, judge); the type of the case (e.g., criminal, civil, multi-district); and the document length (e.g., memoranda are typically long; minute orders typically short). Combining these non-lexical features with text features requires a different classifier than the language-model classifier used in the first set of experiments.
We compared the performance of SupportVector Machine (SVM)
learning (WEKA’s implementation of Platt’s algorithm for sequential
minimal optimization [
As shown in Figure 4, the SVM was consistently more accurate, with little variation in accuracy as a function of the number of features, although accuracy was slightly higher at 4,000 features than other feature set sizes. By contrast, the accuracy of the random forest diminished with increasing numbers of features. The highest accuracy SVM configuration, f-measure of 0.926, was much higher than the maximum observed with text-only classification (albeit, in a different court). This suggests that including procedural context features is essential for accurate document filing type identification in judicial databases, and that algorithms that can handle both textual and categorical features should be used for this task. 4
In many federal courts, docket clerks are responsible for filing orders executed by judges into the docket system, a process that requires the clerk to identify all pending motions to which the order responds and to link the order to those motions. This entails reading all pending motions, a tedious task. If the motions corresponding to an order could be identified automatically, docket clerks would be relieved of this laborious task. Even ranking the motions by their likelihood of being resolved by a given order would decrease the burden on docket clerks. Moreover, order/motion matching is a subtask of a more general issue-chaining problem, which consists of identifying the sequence of preceding and subsequent documents relevant to a given document.
A straightforward approach to this task is to treat order/motion matching as an information-retrieval task, under the hypothesis that an order is likely to have a higher degree of similarity to its corresponding motions than to motions that it does not rule on. An obvious approach is to present pending motions to the clerk in rank order of their TF/IDF2-weighted cosine similarity to the order.
The evaluation above showing that term selection improves document classification raises the question whether term selection might be beneficial for order/motion matching as well. A second question is whether the IDF motion should be trained on an entire corpus of motions and orders or whether acceptable accuracy can be obtained by training just on the order and pending motions.
To evaluate the effectiveness of this approach to order/motion match, a subset of the document set described above was collected consisting of 3,356 groups, each comprising (1) an order, (2) a motion that the order rules on (a triggering motion), and (3) a nonempty set of all motions that were pending at the time of the order but not ruled on by the order (non-triggering motions). The mean number of motions per group was 5.87 (i.e., there were on average 4.87 non-triggering motions). For each group, all motions were ranked by similarity to the order under the given metric. The proportion of triggering motions that were ranked first and mean rank of the triggering motion were calculated from each group’s ranking.
These groups were evaluated using three term selection approaches: the raw document text (which often contains many OCR errors); normalization, as described above; and event terms. The two alternative TF/IDF training models were applied to each of the three term selection approaches, for a total of 6 combinations. For each combination, the mean rank of the triggering motion among all the motions was determined.
Figure 5 shows that the highest accuracy, as measured by the proportion of triggering motions that were ranked first among all pending motions, was achieved by normalizing the text without term selection. Intuitively, reduction to procedurally relevant terms improves the ability to determine what docket event a document performs, but can reduce the ability to discern the similarity between corresponding pairs of documents. TF/IDF training on just the order 2Term Frequency/Inverse Document Frequency and pending motions (local) is at least as accurate as training over all orders and motions (all). Figure 6 shows the mean rank (zero indexed) of the most similar motion under each of the six conditions. The best (lowest) mean rank was achieved with normalization and local TF/IDF training.
It is not unusual for a single order to rule on multiple pending motions. A more realistic assessment of the utility of pending motion ranking is therefore to determine how many non-triggering motions a clerk would have to consider if the clerk read each motion in rank order until every motion ruled on by the order is found. One way to express this quantity is as mean precision at 100% recall. In the test set described above, using text normalization and local TF/IDF training, mean precision at 100% recall was 0.83, indicating that the number of motions that a clerk would have to be read was significantly reduced. 5
Predictive models of decision making could be useful to pro se litigants (for help in understanding the strength of a case), to attorneys (for help in making strategic litigation decisions), and for training and decision support for judges and other decision makers. Even if the accuracy of predictive models were only approximate, they could nevertheless be valuable for decision support by helping to identify the most relevant words, phrases, or other features of a case record and the most relevant previous decisions.
Highly-accurate predictive models would require very detailed
linguistic analysis of the text of case records and decisions, including
argument structure, narrative analysis, etc. [
Unfortunately, in many simple and routine administrative
domains, only the decisions themselves, but not the underlying case
records, are available. However, in such cases, the statement of facts
in the decision can be used as a proxy for the contents of the
corresponding case record. This approach was applied to decisions of
the European Court of Human Rights in [
Accordingly, an experiment was performed to evaluate the feasibility of predicting decisions from the fact statements of cases in representative domain: World Intellectual Property Organization (WIPO) domain name decisions.3 Domain name decisions resolve disputes between a domain name registrant and a third party under the Uniform Domain Name Dispute Resolution Policy (UDRP).4 The UDRP Administrative Procedure applies to disputes concerning an alleged abusive registration of a domain name under the following criteria: • The domain name registered by the domain name registrant is identical or confusingly similar to a trademark or service mark in which the complainant (the person or entity bringing the complaint) has rights; and • The domain name registrant has no rights or legitimate interests in respect of the domain name in question; and • The domain name has been registered and is being used in bad faith
WIPO decisions have a very consistent structure, including sections for History, Background, Contentions, Findings, and Decision. Just two distinct decisions are possible: transferring the domain name or denying the complaint. As a result, the decisions are well suited to the supervised concept-learning approach described above. 5.1
Six thousand six hundred WIPO decisions were downloaded and parsed into the vfie sections described above. Each decision was labeled TRUE or FALSE based on whether the decision transferred the domain name (TRUE) or denied the claim (FALSE). The resulting set of cases had significant class skew, with 6,000 instances of TRUE but only 500 instances of FALSE. For a preliminary study, the 500 random TRUE instances were subsampled to create a balanced test set with 500 instances of each category.
This balanced set of cases was converted into a series of test sets differing in which sections were included as the text of each instance. The sections tested were as follows: • History • Background • Contentions 3http://www.wipo.int/amc/en/domains/decisions.html 4https://www.icann.org/resources/pages/policy-2012-02-25-en • The concatenation of History, Background, and Contentions • Findings The text of each instance was normalized by standardizing case and removing punctuation and, in addition, either (1) removing stop words, or (2) retaining stop words but replacing dates and numbers with standard tokens (“NUMBER" or “DATE"). The test condition in which the text consists of Findings is included for completeness, although it is not a good proxy for the case record as it contains conclusions about the facts.
For each selection of case sections and standardization method, the text was converted into n-gram frequency vectors for n=1–4, with only those n-grams retained that occur at least 8 times. The result was converted into sparse arff format,5 loaded in Weka, and evaluated in 10-fold cross-validation using Weka’s implementation support vector machine (SVM) with sequential minimal optimization.
As set forth in Table 2 and Figure 7, the greatest predictive accuracy was achieved by the combination of the History, Background, and Contentions sections of each case (HBC). The predictive accuracy from these three sections, f-measure of roughly 0.95, was almost as high as the accuracy of prediction based on the text of the Findings section.
To understand why the HBC text is so predictive, it is helpful
to examine the terms with the highest mutual information with the
concept to be predicted, some of which are shown in Figure 8. This
5http://www.cs.waikato.ac.nz/ml/weka/arff.html
excerpt shows that phrases concerning filing, failure to submit a
response, notification, and default are particularly strongly associated
with the outcome of the case. One may view high information-gain
phrases as being similar to the factors in [
WIPO domain name dispute cases may be particularly conducive to predictive modeling owing to their binary outcomes and relatively stereotypical fact patterns. This experiment does not address the differences between the case record and the facts as summarized in the decision document, and the evaluation above artificially diminished the effect of class skew by subsampling to produce a balanced test set. Nevertheless, the impressive accuracy of a predictive model trained on raw text without any feature design or knowledge engineering suggests that this approach may have great promise for increasing access to justice for pro se litigants and improving training and decision support for decision makers in domains with many routine adjudications. 6
The history of applying text classification techniques to legal
documents dates back at least to the 1970s [
There is extensive current activity in predictive models trained
on factors unrelated to the merits of the case such as the nature
of suit, attorneys, forum, judge, and parties [
Judicial document collections contain a rich trove of potential information, but analyzing these documents presents many challenges. This paper has demonstrated how many types of filing error detection can be formulated as text classification problems. The highest accuracy was obtained by combining lexical features that characterize the document itself with procedural context features that indicate the role that the document is intended to play. These results demonstrate the feasibility of automating portions of the process of auditing court submissions, which could significant reduce a persistentdrain on court resources.
The experiment with order/motion matching demonstrates that while term selection may improve accuracy for document classification, it can decrease accuracy for tasks that involve matching based on overall similarity rather than procedural similarity.
The demonstration of outcome prediction in WIPO decisions illustrates that for case corpora with a limited set of possible outcomes and relatively stereotypical fact patterns, decision models of impressive accuracy can be induced without hand-engineered features, simply from the fact descriptions. This approach may be particularly promising for decision support and improved access to justice in the simpler and more routine end of the judicial spectrum.
No single technology is applicable to all judicial documents, nor is any approach sufficient for all document analysis tasks. However, each addition to this suite of technologies adds to the capabilities available to the courts, government agencies, and citizens to exploit the deep well of information latent in judicial document corpora.
The MITRE Corporation is a not-for-profit Federally Funded Research and Development Center chartered in the public interest. This document is approved for Public Release, Distribution Unlimited, Case Number 17-0362. ©2017 The MITRE Corporation. All rights reserved.