Access to justice can increase only if AI tools developed in laboratory settings are implemented at scale to address real-world problems. This paper urges the development of implementation science for AI and law-a set of principles and methods to facilitate and study the transfer of AI techniques and tools to real-world use cases. The paper contributes to this development by reporting a case study using LA-MPS, an innovative software application (Legal Apprentice, or “LA”) for reading, searching, and annotating legal decisions, which currently consists of three integrated web applications: Marker, Pad, and Search. LA-MPS is open-source, free, and adaptable to diferent legal domains, and it is deployable both locally (edge-centric) and through cloud servers. The case study uses decisions issued by the U.S. Board of Veterans' Appeals (BVA) that adjudicate disability benefits. The case study simulates the workflow of standard legal research, conducted on a large dataset of unread but automatically annotated legal decisions (10,003 BVA decisions, containing 1,360,230 sentences). Two primary experiments were conducted. First, we used semantic auto-labeling to filter out a subset of 449 decisions (100,514 sentences) that deal with post-traumatic stress disorder (PTSD), and we evaluated auto-labeling accuracy using a stratified random sample (25 decisions, containing 5,529 sentences). Second, we conducted semantic searches on the set of 449 decisions to identify scenarios in which non-VA evidence prevailed over conflicting VA evidence. The case study is designed to demonstrate the feasibility of implementing currently available machine learning (ML) models at scale, to employ scientific methods to compare results at scale with laboratory results, and to evaluate the real-world results based on practical usefulness. The paper discusses the generalizability of these methods for implementation science. The software code and case study datasets are made available to the public.
plemented at scale to address real-world problems. To
provide high-quality representation for clients, legal aid
organizations should provide their advocates with online
research tools using up-to-date primary sources [
The biomedical field has long recognized a similar need to translate results from laboratory and clinical research man health at scale [3]. After decades of research on the challenges that impede such translation, the biomedical ifeld has evolved ”translational science,” involving operational principles and evidence-informed best practices [3]. The field of AI and law is evolving along a similar path. nEvelop-O LGOBE Proceedings of the Sixth Workshop on Automated Semantic Analysis of pensation [4]. Our case study conducted research on the reasoning in a very large batch of BVA decisions (10,003
We conducted two primary experiments. First, we used semantic auto-labeling to filter out a subset of 449 decisions (100,514 sentences) that deal with post-traumatic stress disorder (PTSD), and we evaluated auto-labeling accuracy using a stratified random sample (25 decisions, containing 5,529 sentences). Second, we conducted semantic searches on the set of 449 decisions to identify and investigate scenarios in which non-VA evidence prevailed over conflicting VA evidence.
In the next three sections of this paper, we describe our case study: Section 2 describes the datasets and predictive model employed; Section 3 provides an overview of the software components; and Section 4 reports the results of the two primary experiments. After that, Section 5 discusses some implications for developing implementation science for AI and law, and Section 6 reviews recent, related work.
The major contributions of this paper are: (primarily stating the content of the testimony of a witness, stating the content of documents introduced into evidence, or describing other evidence); Reasoning Sentences (primarily reporting the trier of fact’s reasoning underlying the findings of fact, which often involves an • a case study evaluating the feasibility of standard assessment of the credibility and probative value of the attorney workflows for argument mining at scale; evidence); Legal-Rule Sentences (primarily stating one or • open-source, adaptable, and free software code more legal rules in the abstract, without stating whether for a suite of integrated web applications designed the rule conditions are satisfied in the case being decided); to assist legal practitioners in performing some Citation Sentences (referencing legal authorities or other main tasks involved in legal research; materials, and usually containing standard notation that • a set of curated BVA decisions that add to well- encodes useful information about the cited source); and known, gold-standard datasets of decisions for Other Sentences (not fitting into any of the previous 5 catclaims involving PTSD; and egories). The number of sentences labeled for sentence type in the LLT Dataset is 6,153, with the frequencies of • a discussion of best practices to promote the im- sentence types shown in Table 1.
plementation of AI tools at scale in law. The FS-PTSD Dataset. To investigate the automatic labeling (“auto-enrichment”) of legal decisions at scale, we 2. Components for the Case Study started with a set of decisions downloaded from the BVA website, the first 10,003 decisions issued in 2018 (the year This section discusses the datasets and the predictive after the decisions included in the LLT Dataset). These model used in the case study. decisions are suficiently anonymized by the BVA before they are issued. We automatically converted these plain2.1. The Three BVA Datasets text decisions to LSJson format (see Section 3.1 below), which resulted in a total of 1,360,230 sentences. We then The LLT Dataset. To establish a baseline and train a ma- used a predictive model trained on the LLT Dataset (see chine learning (ML) classifier for the case study, we used Section 2.2 below) to auto-label these sentences for the 6 a dataset of BVA decisions annotated and made publicly sentence types from the LLT Dataset. available by the Research Laboratory for Law, Logic and Because the enrichment pipeline auto-labeled all Technology (LLT Lab) at the Maurice A. Deane School of 1,360,230 sentences, we could use the semantic autoLaw at Hofstra University (the “LLT Dataset”) [5, 6, 7].1 labeling to filter a subset of decisions of interest [ 8]. To That dataset consists of 50 decisions that adjudicate dis- create a manageable dataset for evaluation, we filtered ability claims filed by veterans for service-related PTSD, the 10,003 auto-enriched decisions for Finding Sentences issued from 2013 through 2017. This dataset is very well- that contained the word “PTSD”, and we generated a studied in the AI and law research community (see Re- count of the number of such sentences per decision. We lated Work, Section 6.2 below). then collected the set of all decisions that contained four
The LLT Dataset labels the six rhetorical roles in legal or more such sentences (the ”FS-PTSD Dataset,” cases = reasoning that sentences might play: Finding Sentences 449, sentences = 100,514).2 We indexed these LSJson files (primarily stating a finding of fact); Evidence Sentences in Elasticsearch (see Section 3.4). 1The dataset is available at: https://github.com/LLTLab/VetClaims- 2The FS-PTSD Dataset is available at: https://github.com/LegalApJSON/BVA Decisions JSON Format. prentice.
enrichment process for this use case, we drew from the Performance Measures for the NN Model on the Test Data of FS-PTSD Dataset a stratified random sample of 25 deci- the LLT Dataset, by Sentence Type sions (the “SRS Dataset,” consisting of 5,529 sentences), stratifying on the number of hits per decision.3 The SRS Precision Recall F1-score Dataset was stratified as follows: 8 decisions with 4 hits Citation Sents 0.98 0.98 0.98 (i.e., decisions containing 4 Finding Sentences that con- Legal-Rule Sents 0.89 0.88 0.89 tain the word “PTSD”), 6 decisions with 5 hits, 6 decisions Evidence Sents 0.88 0.95 0.91 with 6 or 7 hits, and 5 decisions with 8 or more hits. FRienadsionnginSgenStesnts 00..6765 00..7591 00..5787
To focus on the BVA’s reasoning, we evaluated the Other Sents 0.81 0.77 0.79 predictive accuracy of automatic sentence typing only for sentences in the analysis or discussion sections of the BVA decisions (i.e., those sentences occurring within Table 3 the document section headed as “REASONS AND BASES Confusion Matrix for the NN Model on the Test Data of the FOR FINDINGS AND CONCLUSIONS”). These document LLT Dataset, by Sentence Type sections have no set internal structure, and they contain C L-R E F R O All the rationale for any conclusions reached by the BVA.
In total, there were 4,003 such sentences within the SRS Dataset, with the frequencies of sentence types shown in Table 1.
C L-R E F R O All
We wanted to test the usefulness of relatively simple predictive models, which could be trained and deployed Table 4 without cloud computing in situations with security, pri- oPnertfhoermTeasntcDeaMtaeaosfutrheesLfLoTr tDhaetpasree-tt,rbayinSeedntReonBceERTTyapeModel vacy, or other legal concerns. Moreover, we envision user experimentation with new semantic tags for argumen- Precision Recall F1-score tation roles (see Sections 3.2.B and D below), requiring Citation Sents 1.00 0.98 0.99 economical auto-enrichment of large subsets of decisions. Legal-Rule Sents 0.76 0.97 0.85 The software we developed (see Section 3 below) can use Evidence Sents 0.91 0.94 0.93 legal documents that have been semantically enriched Finding Sents 0.78 0.90 0.84 using any predictive models. Reasoning Sents 0.84 0.60 0.70
For this case study, we used the basic neural network Other Sents 0.76 0.66 0.71 (NN) model reported in [9]. When we trained and tested it on the LLT Dataset, we obtained the performance measures on the test data (30%) shown in Table 2. The confu- formation to perform a very diferent task [ 10]. We tested sion matrix for the test set is shown in Table 3 (columns their RoBERTa setup on the LLT Dataset and obtained display actual types, rows display predicted types). We test results that were quite similar to those for our NN discuss the model performance for the SRS Dataset in model, with the exception of better results for Reasoning the case study in Section 4.1. On a laptop, it took about Sentences (compare Tables 2 and 4, row 5). For our case 24 hours to convert the 10,003 decisions to LSJson and to study, the pre-trained RoBERTa model would have made auto-enrich the resulting 1,360,230 sentences. little if any diference (see Section 4 below).
The present case study is not designed to improve model performance, but to test the practical utility of using auto-enrichment at scale. When newer models are 3. LA-MPS Overview developed, they are not always substantial improvements from a practical standpoint. For example, researchers have reported that a pre-trained RoBERTa model had outperformed other models on a diferent dataset of legal decisions, which were labeled with diferent semantic in
vide user interfaces (UIs) for reading, annotating, and querying sets of semantically enriched legal documents. Legal Apprentice currently has three separate web applications that communicate with each other: LA-Marker,
tice. network model not performing significantly better than the CRF model [12]. Occasional errors in converting individual documents can be corrected using the Editor Mode of LA-Marker (see Section 3.2(F) below).
The LA-MPS software stores, processes, and exchanges data in JSON format. The format that we call Legal Semantic JSON (“LSJson”) is a lightweight, extensible, dataexchange format for capturing the text and the semantic information associated with legal documents. LSJson stores the original decision string of characters, metadata about the decision, details about any predictive models used to auto-enrich the document, and added semantic information about the sentences and paragraphs of the decision. Decisions from a legal tribunal must first be converted from their original formats into LSJson. We have successfully converted original decision files from plain-text, HTML, and PDF formats.
The conversion package for LA-MPS is separate from the rest of the code, and any adequately accurate conversion package can be used, whether rule-based or based on machine learning. Researchers have demonstrated that general systems for detecting sentence boundaries perform much worse on legal documents when compared to their performance on news articles data sets [11], and they showed that a general conditional random ifeld (CRF) model trained on the legal data performed significantly better. Later research confirmed that the CRF model is the most practical approach, with a neural 4The software code is open source and free, and it is available at: https://github.com/LegalApprentice. • Displaying the semantic information stored in the LSJson file of the document (e.g., documentlevel metadata, or sentence-level types, notes, and tags). • Filtering the document by six sentence types (discussed in Section 2.1): Finding, Evidence, Legal Rule, Reasoning, Citation, and Other (Figure 2, at (2)). Filter buttons display lists of sentences of the selected type in the order in which they occur in the document (Figure 2, at (4)). • Manually adding or editing the type (role) of a sentence. If an automatic classifier has been used to predict a sentence’s type, all possible types are displayed in buttons showing their predicted classification scores. Selecting a button manually assigns a rhetorical role to the sentence. • Manually adding or editing sentence-level notes or tags, as well as paragraph-level notes or tags (see Figure 2, at 3)). • Selecting sentences or paragraphs for further action (see the Selections Mode below).
(1)) displays entire paragraphs in order of priority. The lists are currently sorted by an “interest-score” devised to prioritize paragraphs that might contain entire arguments. The user can add or edit paragraph-level notes or tags, or select specific paragraphs for further action (see Selections Mode below).
D. Notes/Tags. Notes/Tags is a user mode that displays, in grid format, all the notes or tags (sentence-level or paragraph-level) that are present in the document. A database of notes or tags can be exported as a CSV file.
E. Selections. The Selections Mode allows a user to but it could be configured to use any search engine. Figgather selected sentences and paragraphs, so the user ure 4 shows a view of Search which displays a selection can send the selections to LA-Pad for further annotation of sentences that can be sent to Pad for further semantic (discussed in Section 3.3 below). enrichment (see Section 3.3). The content of the example
F. Editor. In working with documents in LSJson format, in Figure 4 will be discussed in Section 4.2 below, where the user might occasionally find errors in segmentation or we discuss the results of the case study. boundary identification (e.g., of sentences or paragraphs).
The user can correct such errors in the LSJson file itself, not merely in the display.
3.3. LA-Pad LA-Pad (or simply Pad) is a web application with a UI for gathering sentences and paragraphs from diferent decisions (either via Marker or via Search), grouping them into user-defined sets, and annotating those groups with notes or tags. Pad is designed to simulate a traditional “legal pad,” which a lawyer might use to gather notes on a topic of interest as her research proceeds. Pad files can be saved and reopened later, to add further research. Figure 3 shows a group of sentences sent either from Marker (see Section 3.2) or from Search (Section 3.4), in the UI for grouping and annotating selected items. The contents displayed in Figure 3 will be explained in Section 4.2, where we discuss the results of the case study.
Using LA-MPS, we conducted a case study simulating normal attorney workflows. We evaluated the accuracy of the auto-labeling in the SRS Dataset for reading annotated decisions, and we performed typical legal research tasks using the auto-enriched FS-PTSD Dataset. This section discusses our results.
reading a legal decision to identify the elements of reasoning patterns from the evidence to the findings of fact.
We used LA-Marker to read annotated decisions and to assess the likelihood and practical significance of errors in sentence auto-labeling.
Quantitative Performance Metrics. The evaluation of the adequacy of the auto-labeling occurred at two levels in the case study. First, in the SRS Dataset, all 25 decisions in fact decided claims involving PTSD. Nine of these either granted or denied claims to establish a ser- sum, there were no false positives in the SRS Dataset at vice connection for PTSD, and 14 either granted or denied the level of whole decisions. claims for a particular disability rating for PTSD (e.g., a Second, to evaluate the accuracy of the autoclaim to set the disability rating at 70 percent). Disability enrichment, a practicing attorney with expertise in legal ratings (expressed as a percentage) are assigned to vet- reasoning constructed for each of the 25 decisions in the erans based on the severity of their disability. Disability SRS a confusion matrix, and we calculated the perforratings are used to determine a disability compensation mance of the predictive model on the SRS Dataset. The rate, and they help determine eligibility for other vet- performance measures for the auto-labeling of sentences erans benefits. A few decisions addressed a variety of in the SRS are shown in Table 5, and the overall confuother PTSD-related claims (e.g., claiming disability for sion matrix is shown in Table 6 (columns display actual sleep apnea secondary to service-connected PTSD). In types, rows display predicted types). The results are reTable 5 Dataset focused on claims to establish service connection Performance Measures for the NN Model on the Analysis for PTSD in the first instance, many of the SRS decisions Sections of the SRS Dataset, by Sentence Type were claims involving a particular disability rating for PTSD. Within the SRS decisions to establish a rating, one
Precision Recall F1-score recurring error was auto-labeling as Evidence Sentences Citation Sents 0.97 0.96 0.96 the detailed regulatory criteria and symptoms for assignLegal-Rule Sents 0.93 0.57 0.71 ing a particular percent of disability. Because such senEvidence Sents 0.90 0.94 0.92 tences report symptoms from regulations, they should be FRienadsionnginSgenStesnts 00..8741 00..7523 00..7681 labeled as Legal-Rule Sentences, not Evidence Sentences. Other Sents 0.56 0.96 0.71 Such classification errors occurred for 55 sentences in the SRS Dataset. If the model is re-trained on a dataset that includes such sentences as part of the gold standard, then Table 6 such auto-labeling errors might be reduced or avoided. Confusion Matrix for the NN Model on the Analysis Sections Practical Cost of Error. The confusion matrix for the of the SRS Dataset, by Sentence Type SRS Dataset also suggests that even when false positives C L-R E F R O All occur, they are often of little practical importance given our use case. For example, when a sentence is mislabeled C 707 11 4 2 2 3 729 as a Finding Sentence, it may actually be a Reasoning L-R 2 353 1 14 10 0 380 Sentence (see Table 3, row 4; Table 6, row 4). The pracEF 03 989 116165 11798 1527 48 1281531 tical cost of such an error is likely to be low because R 2 35 9 15 149 0 210 a search result involving a Reasoning Sentence instead O 25 110 72 18 49 346 620 of a Finding Sentence could still produce an instructive All 739 616 1762 246 279 361 4003 example of reasoning on the search topic.
In addition, it became clear in the case study that in a LA-MPS work environment, many sentence-level typing errors are in practice “harmless errors” because they are visually obvious. When reading a decision in which an entire paragraph is devoted to reciting evidence or legal rules (as is often the case), an isolated classification error for a single sentence or two stands out visually, and the error is quickly recognized and discounted mentally. Another example involves sentences auto-labeled as Other Sentences, a category with much lower precision than other categories (see Tables 5 and 6, row 6). In LA, Other Sentences are color-coded with white background.
When reading a decision in Marker, such sentences tend to stand out visually and be mentally re-classified by the user. Thus, the eficiency with which LA-Marker allows an auto-enriched decision to be read and understood is not much afected by visually obvious errors.
In sum, this part of the case study confirms our hypothesis that a predictive model trained and tested on a relatively small dataset (the LLT Dataset) can retain its level of performance when used to auto-enrich at scale.
Also, some errors can be visually discounted in practice when reading decisions with semantic color-coding. markably similar to the results in the test set of the LLT Dataset (compare Tables 2 and 5). For BVA decisions, these results were achieved with a very small training set and a relatively small training vocabulary.
In particular, the false positive rate for Finding Sentences in the SRS Dataset (precision = 0.84, for a false positive rate = 0.16) was lower than we observed when we trained the model on the LLT Dataset (false positive rate = 0.25). Finding Sentences are critical in identifying reasoning because they state the conclusions of the tribunal. The precision measures for Evidence Sentences, Legal-Rule Sentences, and Citation Sentences were quite high (0.90, 0.93, and 0.97, respectively). Reasoning Sentences had comparable F1-scores in the LLT and the SRS datasets, and they had the least accuracy of all sentence types.
These results at scale were consistent with data-centric analysis using the LLT Dataset [13], which suggested that this classification system of sentence types would be robust as the number of labeled BVA decisions increases.
The present case study provides some confirmation that a data-centric analysis can be indicative of robustness of a classification system at scale.
Moreover, many of the prediction errors we observed in the SRS Dataset are perhaps avoidable through retraining the model by combining the LLT Dataset with a curated SRS Dataset.5 For example, while the LLT
it is important to identify the relevant decisions to read.
For this aspect of the case study, we focused on resolving 5The curated SRS Dataset is available at: https://github.com/LegalAp- conflicting evidence. prentice. Indexing the 100,514 sentences from the FS-PTSD It can be a challenge to identify all the normal workflows involved in legal work at scale. Our case study investigated reading annotated sentences in context and filtered for sentence types, as well as searching for relevant decisions to review. Other workflows might include questionanswering (employing chatbots), document summarization, document drafting, and predictive-factor extraction.
It would be worthwhile to identify all the security, privacy, and other regulatory constraints on working with legal data at scale. Our case study simulated a scenario in which the labeled data, model development, and auto-enrichment would all be local. Depending upon the constraints in a particular implementation, cloud storage and computing could be used instead.
Implementation at scale places a premium on the validity of the gold-standard data and of the results, not merely on the reliability. Given the resource-intensive nature of generating data for training and testing, it can be more eficient to employ non-experts to manually label data or to evaluate the predictive results, and to rely upon measures of reliability (consistency) among labelers for quality assurance. In real-world implementation, however, the validity (accuracy) of the labeling is what is important—i.e., whether the text is correctly classified.
Our case study experiments employed a practicing attorney with expertise in legal reasoning. What are needed are best practices for evaluating validity at scale, as eficiently as possible.
Accuracy is always a concern when transferring predictive models from laboratory settings to auto-enrichment at scale, especially when the training and testing has been done on a small dataset that the model might have overfit. Random sampling from a population at scale can provide some reassurance about continued accuracy after model transfer. What are needed are best practices for using random samples to estimate statistics like precision, recall, and F1-scores for data populations at scale.
Making both open-source code and labeled data publicly available is important for facilitating widespread implementation at scale. But also, when the objective is the real-world validity of the results, replication and veriifcation are necessary to achieve a consensus on accuracy at scale.
Implementing a use case at scale poses the challenge of evaluating its practical usefulness. For example, a moderate level of performance, with precision on the order of 0.75 and similar recall, might be adequate for some practical use cases—such as in the use case reported also performed a qualitative error analysis on a sample in this paper, mining illustrative examples of arguments of 200 erroneous predictions, concluding that “even inthat have been successful in certain evidential situations. correct predictions may still be useful.” One next step, In our case study, we have tried to evaluate the results they concluded, was ”to evaluate the usefulness of the using both standard quantitative measures and a practical models trained here with expert users.” assessment of the cost of expected errors. The second study simulated drafting extractive summaries that could enable readers to make an informed decision about whether to read the full decision [15]. 6. Prior Related Work From nearly 1 million BVA decisions, researchers filtered about 35,000 single-issue decisions that dealt with serThis section surveys recent related work in three areas: vice connection for PTSD, from which they randomly research on auto-enrichment and evaluation at scale, re- sampled 112 cases for their experiments (92 for training search using BVA datasets, and other recent innovative and validation, 20 for testing). The task was to generate applications. case summaries that were between 6-10 sentences long, in which 2-6 sentences should summarize the BVA’s rea6.1. Auto-Enrichment and Evaluation at sons and the evidence considered. They first extracted Scale ”predictive sentences,” and then they trained a random forest classifier to classify them as either “Reasoning/EvThere have been many experiments at scale on individual identialSupport” or “Other.” For 954 training sentences tasks related to legal research. To the best of our knowl- and 341 testing sentences, their classifier reached 0.85 edge, none have deployed integrated web applications to precision, 0.77 recall, and 0.81 F-score. They used the assist standard attorney workflows in reading, searching, sentence classification and Maximal Marginal Relevance and annotating legal decisions enriched at scale for se- (MMR) to select the variable number of Reasoning/Evimantic information, the workflows assisted by LA-MPS. dential Support sentences for the summary. Although We discuss here several recent studies that have evalu- their ROUGE-1 and ROUGE-2 scores were only 0.269 ated auto-enriched samples drawn from large sets of legal and 0.102, respectively, they had evidence from human decisions. We discuss them from the perspective of the drafted summaries that the value of ROUGE scores as three principles of implementation research discussed in metrics were of limited use for evaluating summaries of Section 5. legal opinions. They also conducted extensive qualitative
Two studies have used the extensive corpus of full- error analysis, from which they hypothesized that “sentext BVA decisions (over 1 million decisions from 1999 to tences involving evidential reasoning” might be useful 2017 [14]) to conduct research at scale. One advantage of for identifying more details in automated summaries. using this corpus is that decisions are anonymized before Other recent large-scale research includes: explorthey are published. ing court data, using more than a quarter-million case
One study simulated a BVA staf attorney drafting an dockets in HTML format and ontology-leveraged tools opinion in a new case, and the study developed a tool [8]; predicting the outcome of motions on the basis of to recommend a legal citation to a published judicial de- court administrative data and complaint documents, uscision, statute, or regulation [14]. From over 1 million ing 184,125 civil cases from the State of Connecticut JuBVA decisions, researchers filtered a subset of 324,309 dicial Branch to draw a sample of 7904 motions to strike, BVA cases that raised a single issue and had complete and testing 6 auto-classification models [ 16]; predicting metadata. They split those cases into training, validation, verdict labels on the basis of the pre-verdict text of a and test sets (72%, 18%, 10%, respectively). Two neural decision, using a corpus of 544,857 court decision documodels (a Bi-directional Long Short Term Memory model ments in French for landlord-tenant disputes in Quebec, and a fine-tuned RoBERTa-based model) performed com- Canada, and CamemBERT (a BERT model pretrained on parably and better than other methods, using sequences French material) [17]. of words in the draft opinion as context to predict the next citation. They used recall at 1, 5, and 20 as the quantitative metric (the proportion of data instances where 6.2. Research Using BVA Datasets the correct next citation is among the model’s top 1, 5, In addition to the two BVA studies at scale discussed in or 20 predictions). They considered recall@5 to simu- Section 6.1, researchers have employed the small LLT late a “typical user, who benefits from a small number of Dataset of 50 BVA decisions in various studies. recommendations that can quickly be examined for the Some have tested methods and tools to perform tasks most appropriate.” Neural model training for extended that could be relevant to enriching BVA decisions at scale. periods continuously improved up to a recall@5 of 83.2%, Recent studies include: training a general conditional which they considered “acceptable performance.” They random field (CRF) model to detect sentence boundaries [11, 12]; classifying Finding Sentences for their linguistic system with a natural UI connected to an Abstract Dipolarity (i.e., whether the finding is positive or negative alectical Framework (ADF) to predict admissibility before on the legal issues presented) [9]; evaluating an anno- the European Court of Human Rights, and to present an tation system (CAESAR) based on the hypothesis that explanation of the prediction to the user [27]; and a platsentences that are semantically similar often have the form that allows users to explore data and drive analysis same rhetorical type [18]; investigating how changes in by leveraging an ontology configuration, with natural the size of the dataset, the train/test splits, and human la- language statements in the UI [8]. beling accuracy afect the performance of a trained deep learning classifier [ 13]; and testing whether a small set of labelled data could train deeper and more accurate 7. Conclusion and Future Work predictive models (obtaining the highest accuracy with a Bidirectional Long Short-Term Memory (Bi-LSTM) model for classifying sentence rhetorical roles) [19].
Others have investigated methods and tools for performing related tasks, such as: assessing the performance of diferent explainability methods (XAI), after using a convolutional neural network for classifying sentences for rhetorical role [20]; identifying factors that the tribunal considers when assessing the credibility or trustworthiness of individual items of evidence [21]; investigating computer-assisted text classification using Boolean matching rules (CASE) [22]; and examining the ability of pre-trained language models to generalize beyond the legal domain and dataset they were trained on (finding that the performance of an SVM and a RoBERTa model trained to classify sentences as “fact” or “non-fact” was “surprisingly high,” despite being trained on datasets from diferent domains and jurisdictions) [ 23].
plications, LA-MPS, to assist attorneys and judges when reading, searching, and annotating semantically enriched legal decisions. Some notable innovative applications in recent years could assist legal practice with some aspects of this workflow.
Recent applications for annotation include: the Scribe web application for annotation of French court decisions, which facilitates a collaborative workflow between annotators and developers [24]; a LegAi annotation editor, which supports annotating legal texts with the LegAi higher language, with the goal of constructing formal knowledge bases that can support eficient reasoning [25]; an annotator assistant that allows users to create, update, and delete annotations suggested by an algorithmic annotator for named entity recognition (NER) [26]; and a prototype interface CAESAR (Computer-Assisted Enhanced Semantic Annotation & Ranking) that allows assigning the same rhetorical type to sentences that are semantically similar [18].
Other use cases include: an intelligent tutoring system for analyzing legal decisions, employing a cognitive computing framework that matches various ML capabilities to the proficiency of the user [ 10]; a legal support
and for adaptation to other legal domains. It should be especially useful in legal areas with a high volume of decisions that assess multiple kinds of evidence and employ complicated reasoning, but where mining that reasoning is dificult.
Although the case study reported here examined the usefulness of auto-labeling only for sentence rhetorical role, we have already auto-enriched all Finding Sentences in the 10,003 cases for their linguistic polarity (positive or negative) [9], and for the legal issue addressed by the sentence. This should assist creating a library of legal reasoning patterns at greater granularity (e.g., successful and unsuccessful arguments on specific legal issues).
The LA-MPS environment built on LSJson is also extendable by adding new web applications. Currently under development for LA-MPS is a fourth integrated web application, LA-Draw. This application will enable the user to create graphic conceptual networks connecting terms, sentences, paragraphs, or decisions.
Next steps in the BVA domain include comparative testing of a variety of ML algorithms and large language models, using the combined LLT and SRS curated datasets.
The best-performing models could then be used to autoenrich a larger number of BVA decisions, and to provide an indexed database available for research by the public, by veterans’ representatives, and by lawyers and judges at the BVA. Such a service could be hosted on a cloud computing platform such as Microsoft Azure.
Finally, LA-MPS can also be implemented as the user interface for deploying other tools, such as citation recommendation [14] or auto-generated extractive summaries [15]. In addition, LA-MPS could be used to provide training and quality assurance tools to assist human authors in writing legal decisions, by providing feedback on how well the decision states the tribunal’s reasoning [10]. Adequate Resources for Research and Investigation, citation recommendation using deep learning, in: 2021. Proceedings ICAIL ’21, ACM Digital Library, 2021, [3] J. M. Faupel-Badger, A. L. Vogel, C. P. Austin, J. L. pp. 79–88.
Rutter, Advancing translational science education, [15] L. Zhong, Z. Zhong, Z. Zhao, S. Wang, K. D. Ashley, Clin. Transl. Sci. 15 (2022) 2555–2566. doi:1 0 . 1 1 1 1 / M. Grabmair, Automatic summarization of legal c t s . 1 3 3 9 0 . decisions using iterative masking of predictive sen[4] Board of Veterans’ Appeals, U.S. Department of tences, in: Proceedings of ICAIL ’19, ACM Digital Veterans Afairs, Annual Report, Fiscal Year 2021, Library, 2019, pp. 163–172.
2021. [16] D. J. McConnell, J. Zhu, S. Pandya, D. Aguiar, Case[5] V. R. Walker, K. Pillaipakkamnatt, A. M. Davidson, level prediction of motion outcomes in civil litigaM. Linares, D. J. Pesce, Automatic classification tion, in: Proceedings of ICAIL ’21, ACM Digital of rhetorical roles for sentences: Comparing rule- Library, 2021, pp. 99–108. based scripts with machine learning, in: Proceed- [17] O. Salaün, P. Langlais, K. Benyekhlef, Labels disings of the Third Workshop on Automated Semantic tribution matters in performance achieved in legal Analysis of Information in Legal Text (ASAIL 2019), judgment prediction tasks, in: Proceedings of ICAIL 2019. ’21, ACM Digital Library, 2021, pp. 268–269. [6] V. R. Walker, J. H. Han, X. Ni, K. Yoseda, Semantic [18] H. Westermann, J. Savelka, V. R. Walker, K. D. Ashtypes for computational legal reasoning: Proposi- ley, K. Benyekhlef, Sentence embeddings and hightional connectives and sentence roles in the veter- speed similarity search for fast computer assisted ans’ claims dataset, in: Proceedings of ICAIL ’17, annotation of legal documents, in: Proceedings of ACM Digital Library, 2017, pp. 217–226. JURIX ’20, 2020, pp. 164–173. [7] V. R. Walker, A. Hemendinger, N. Okpara, T. Ahmed, [19] R. Ahmad, D. Harris, M. I. Sahibzada, UnderstandSemantic types for decomposing evidence assess- ing legal documents: Classification of rhetorical ment in decisions on veterans’ disability claims for role of sentences using deep learning and natural ptsd, in: Proceedings of the Second Workshop on language processing, in: Proceedings of the 14th Automated Semantic Analysis of Information in IEEE International Conference on Semantic ComLegal Text (ASAIL 2017), 2017. puting (ICSC2020), 2020, pp. 464–467. [8] A. Paley, A. L. L. Zhao, H. Pack, S. Servantez, R. F. [20] Łukasz Górski, S. Ramakrishna, Explainable artifiAdler, M. Sterbentz, A. Pah, D. Schwartz, C. Barrie, cial intelligence, lawyer’s perspective, in: ProceedA. Einarsson, K. Hammond, From data to informa- ings of ICAIL ’21, ACM Digital Library, 2021, pp. tion: Automating data science to explore the U.S. 60–68. court system, in: Proceedings of ICAIL ’21, ACM [21] V. R. Walker, D. Foerster, J. M. Ponce, M. Rosen, Digital Library, 2021, pp. 119–128. Evidence types, credibility factors, and patterns or [9] V. R. Walker, S. R. Strong, V. E. Walker, Automating soft rules for weighing conflicting evidence: Arguthe classification of finding sentences for linguistic ment mining in the context of legal rules governing polarity, in: Proceedings of the 2020 Workshop evidence assessment, in: Proceedings of the 5th on Automated Semantic Analysis of Information in Workshop on Argument Mining, 2018, pp. 68–78.
Legal Text (ASAIL), 2020. [22] H. Westermann, J. Savelka, V. R. Walker, K. D. Ash[10] H. Westermann, J. Savelka, V. R. Walker, K. D. Ash- ley, K. Benyekhlef, Computer-assisted creation of ley, K. Benyekhlef, Toward an intelligent tutoring Boolean search rules for text classification in the system for argument mining in legal texts, in: Pro- legal domain, in: Proceedings of JURIX ’19, 2019, ceedings of JURIX ’22, 2022, pp. 133–142. pp. 123–132. [11] J. Savelka, V. R. Walker, M. Grabmair, K. D. Ash- [23] J. Savelka, H. Westermann, K. Benyekhlef, Crossley, Sentence boundary detection in adjudicatory domain generalization and knowledge transfer in decisions in the United States, TAL 58 (2017) 21–45. transformers trained on legal data, in: Proceed[12] G. Sanchez, Sentence boundary detection in legal ings of the 2020 Workshop on Automated Semantic text, in: Proceedings of the Natural Legal Language Analysis of Information in Legal Text (ASAIL), 2020.
Processing Workshop 2019, 2019, pp. 31–38. [24] S. A. Mahmoudi, G. Zambrano, C. Condevaux, [13] H. Westermann, J. Savelka, V. R. Walker, K. D. Ash- S. Mussard, Scribe: A specialized collaborative tool ley, K. Benyekhlef, Data-centric machine learning: for legal judgment annotation, in: Proceedings of Improving model performance and understanding JURIX ’22, 2022, pp. 290–293. through dataset analysis, in: Proceedings of JURIX [25] T. Libal, The LegAi editor: A tool for the construc’21, 2021, pp. 54–57. tion of legal knowledge bases, in: Proceedings of [14] Z. Huang, C. Low, M. Teng, H. Zhang, D. E. Ho, JURIX ’22, 2022, pp. 286–289.
M. S. Krass, M. Grabmair, Context-aware legal [26] Y.-T. Huang, H.-R. Lin, C.-L. Liu, Toward an integrated annotation and inference platform for enhancing justifications for algorithmically generated legal recommendations and decisions, in: Proceedings of JURIX ’22, 2022, pp. 281–285. [27] K. Atkinson, J. Collenette, T. Bench-Capon, K. Dzehtsiarou, Practical tools from formal models: The ECHR as a case study, in: Proceedings of ICAIL ’21, ACM Digital Library, 2021, pp. 170–174.