Automated argument mining would add greatly to productivity in legal practice, and it could increase access to justice in many legal areas through a range of use cases. As a first use case, a web-based semantic viewer might automatically highlight for the user those sentences and patterns that are of interest in argumentation. For example, a semantic viewer might open a new decision document and provide filters that the user could select to highlight only the conclusions, or the evidence, or the stated reasoning from evidence to conclusions.

Second, we could create semantic search tools, which would use components of reasoning to search through hundreds of thousands of decisions in plain-text format, retrieve those decisions similar to a new case (e.g., those with similar issues to be proved, and similar types of evidence available for proving them), rank the similar decisions in order of greatest similarity, and then display the portions to read (using a semantic viewer).

Third, the capability of extracting from published decisions the major conclusions, the intermediate reasoning, and the evidentiary basis for a decision would also provide the components of an informative summary of that decision. A decision summarizer could analyze a decision and create a digest or summary for the human reader.

Fourth, an automatic argument miner could extract successful and unsuccessful patterns of reasoning from thousands of past decisions, and then it could generate suggestions for new arguments in new cases. Such an argument recommender could assist attorneys, non-attorneys, and judges in processing new cases.

Fifth, such an automated argument miner could monitor cases as they are being litigated, compare evolving arguments with patterns of reasoning that have been successful or unsuccessful in the past, and detect possible outliers (arguments likely to fail or decisions likely to be incorrect). A reasoning monitor could also maintain statistics and trends for patterns of reasoning, and it could predict probabilities of success for new cases.

Provided the data derived from argument mining is valid and predictive of real-case outcomes, such tools could assist alternative dispute resolution and increase efficiency within the legal system. Such automated, evidence-based tools (semantic viewers, semantic search, decision summarizers, argument recommenders, and reasoning monitors) could also assist non-lawyers when they represent themselves in cases where a lawyer is not available.

Producing such a range of tools, however, faces several challenges. One challenge is whether machine learning (ML) tools can be effective at automating such argument mining. First, there is the problem of the available quantity of gold-standard data for training and testing. Supervised ML may require such a large quantity of accurately labeled documents that there are not sufficient resources to generate it, in all areas of law where argument mining is desirable. While semi-supervised ML, using large quantities of unlabeled data and small quantities of labeled data, offers more promise, even that approach requires trained human classifiers. Especially in legal areas where outcomes may not economically support the hiring of a lawyer (e.g., veterans’ disability claims or immigration asylum claims), there may be little financial incentive to create such a great quantity of ground-truth annotated data. Moreover, some areas of law (such as vaccineinjury compensation decisions) may not even produce a sufficient quantity of cases bearing on a particular issue, even if we were to annotate it all.

Moreover, there is the challenge of ensuring data validity. In order to create effective tools, especially tools that predict future outcomes given past decisions, the data upon which the tools are trained must accurately reflect what we believe it measures. But appropriately annotating the components of legal reasoning requires an adequate theory of legal reasoning, a sufficient number of trained annotators, and adequate quality assurance. Also, to inspire trust in ML outputs, the models must be transparent and understandable.

Finally, there is the challenge of developing and testing adequate classification type systems for legal arguments [ 34, 38 ]. Unsupervised ML has the challenge of producing useful clusters, especially if the components of legal argument are poorly understood, the components vary depending on the use case, and the components are different for different practitioners. How to classify argument types is therefore a problem that any effort to create gold-standard data must address.

This paper reports on preliminary research that addresses all of these problems. Our main hypothesis is that reports of fact-finding in legal adjudicatory decisions might employ such regular and distinctive phrasing that even rule-based scripts based on a very small sample, as well as ML models trained on larger samples, can perform adequately for many valuable use cases. Our second hypothesis is that attribution theory from linguistics can be extended to argument mining, as a method for creating semantic types and automatically identifying them in legal decisions. There is reason to think that such an approach will be transferable to adjudicatory decisions in many substantive areas of law.

Using an annotated dataset of U.S. decisions as the gold standard, we investigated a methodology for qualitatively studying a very small sub-sample of such decisions, developing rule-based scripts, and quantitatively testing the script performance. We compared those outcomes against the performance of standard supervised ML models trained on larger samples from the same dataset. Our study addresses data quantity by employing very small datasets; it addresses data validity by employing quality-assurance protocols and publishing those protocols and the resulting data; and it addresses annotation type systems by explaining their derivation. The paper reports promising results relative to important use cases, and it lays out a methodology that should be transferable to many areas of law at a relatively small cost – thus helping to improve access to justice. We make publicly available the annotated dataset, the quality-assurance protocols, the scripts and the ML settings, at https://github.com/LLTLab/VetClaims-JSON.

After we discuss prior related work in Section 2, we describe our dataset and how we used it in both our script and ML experiments (Section 3). Section 4 describes the qualitative-study experiments and their results, while Section 5 describes the ML experiments and their results. Section 6 contains our general discussion of these combined results and our future work.

The context for the work reported in this paper is the goal of automated argument mining from adjudicatory legal decisions. Such argument mining would automatically extract the evidence assessment and fact-finding reasoning found in adjudicatory decisions, for the purpose of identifying successful and unsuccessful units of evidentiary argument. Researchers generally identify an argument as containing a conclusion or claim, together with a set of one or more premises. [ 21, 40, 30, 15, 32 ]

A first level of analysis is to classify the rhetorical roles of sentences for argument mining – that is, assigning sentences roles as either premise or conclusion. Prior work on classifying such rhetorical roles in adjudicatory decisions includes: applying machine learning to annotate sentence types in vaccine-injury compensation decisions [ 3, 4, 5, 10 ]; assigning rhetorical roles to sentences in Indian court decisions [ 27 ]; classifying sentences as argumentative in the Araucaria corpus, including newspapers and court reports [ 19 ]; automatically summarizing legal judgments of the U.K.’s House of Lords, in part by classifying the rhetorical status of sentences [ 13 ]; annotating sentences as containing legal principles or facts in common-law reports [ 29 ]; and using statistical parsing as the input for computing quasi-logical forms as deep semantic interpretations of sentences in U.S. appellate court decisions [ 17 ]. Al-Abdulkarim et al. provide one overview of statement types involved in legal reasoning in cases, from evidence to a verdict [ 1 ]. The approach we describe in this paper utilizes a type system of rhetorical roles developed to annotate any factfinding decision, and we compare script and ML classifiers for the rhetorical roles of sentences. Moreover, it is not common for datasets to be publicly available, together with protocols for data generation, scripts and codes, to enable confirmation of data accuracy and replication of results.

A recent article compared two experiments in automated classification of legal norms from German statutes, with regard to their semantic type: (1) a rule-based approach using hand-crafted pattern definitions, and (2) an ML approach. [ 39 ] (For similar work on Dutch laws, see [ 16 ].) The performance metrics for the two experiments were comparable on a dataset of manually-labeled statements. While this study is highly relevant to our work, there are distinct differences. We develop a qualitative methodology for developing classification features of sentences in adjudicatory decisions (not statutes), according to their rhetorical role (not norm type), for the purpose of automated argument mining. Our methodology is general, and it should be transferable to adjudicatory decisions in any substantive area of law.

To identify the rhetorical roles of sentences, we employ an extension of the semantic theory of attribution analysis. Attribution, in the context of argument mining, is the descriptive task of determining which actor is asserting, assuming or relying upon which propositions, in the course of presenting reasoning or argument. Although attribution is a classic problem area in natural language processing generally [ 7, 14, 22, 23 ], there has been limited work on attribution in respect to argument mining from legal documents. Grover et al. reported on a project to annotate sentences in House of Lords judgments for their argumentative roles [ 11 ]. Two tasks were to attribute statements to the Law Lord speaking about the case or to someone else (attribution), and to classify sentences as formulating the law objectively vs. assessing the law as favoring a conclusion or not favoring it (comparison). This work extended the work of [ 31 ] on attribution in scientific articles. A broader discussion of attribution within the context of legal decisions is found in [ 34 ]. Unlike the adjudicatory decisions used in our study, the House of Lords judgments studied by [ 11 ] treated facts as already settled in the lower courts. Our study appears to be unique in using attribution analysis to help classify the rhetorical roles of sentences in the evidence assessment portions of adjudicatory texts.

We have also developed classification protocols (classification criteria and methods) for each rhetorical role. We use protocols to give precise meaning to the semantic type, to train new annotators, and to review the accuracy of human annotations. We also use such protocols to guide the development of the features or rule-based scripts for automatically classifying legal texts (e.g., [ 28 ]). Stab and Gurevych have classified such features into 5 groups [ 30 ]. For example, the main verb of a finding sentence tends to be in present tense, while the main verbs of evidence sentences tend to be in past tense. Features derived from the protocols can drive the application of high-precision / low-recall techniques of the kind used successfully by [ 15 ], which we argue is the performance desired for certain use cases but not others.

We developed a common dataset to use in comparing the classification performance of rule-based script classifiers with the performance of ML models. This section describes that dataset and how it was used.

We analyzed 50 fact-finding decisions issued by the U.S. Board of Veterans’ Appeals (“BVA”) from 2013 through 2017 (the “PTSD dataset”). We arbitrarily selected those decisions from adjudicated disability claims by veterans for service-related post-traumatic stress disorder (PTSD). PTSD is a mental health problem that some people develop after experiencing or witnessing a traumatic event, such as combat or sexual assault. Individual claims for compensation for a disability usually originate at a Regional Office (“RO”) of the U.S. Department of Veterans Affairs (“VA”), or at another local office across the country [ 2, 20 ]. If the claimant is dissatisfied with the decision of the RO, she may file an appeal to the BVA, which is an administrative appellate body that has the authority to decide the facts of each case based on the evidence [ 20 ]. The BVA must provide a written statement of the reasons or bases for its findings and conclusions, and that statement “must account for the evidence which [the BVA] finds to be persuasive or unpersuasive, analyze the credibility and probative value of all material evidence submitted by and on behalf of a claimant, and provide the reasons for its rejection of any such evidence.” Caluza v. Brown, 7 Vet. App. 498, 506 (1995), aff’d, 78 F.3d 604 (Fed. Cir. 1996).

The veteran may appeal the BVA’s decision to the U.S. Court of Appeals for Veterans Claims (the “Veterans Court”) [ 20 ], but the standard of review for issues of fact is very deferential to the BVA. In order to set aside a finding of fact by the BVA, the Veterans Court must determine it to be “clearly erroneous.” [ 20 ] And although either the claimant or the VA may appeal a Veterans Court decision to the U.S. Court of Appeals for the Federal Circuit, the Federal Circuit may only review questions of law, such as a constitutional challenge, or the interpretation of a statute or regulation relied upon by the Veterans Court. [ 2, 20 ] Except for constitutional issues, it “may not review any ‘challenge to a factual determination’ or any ‘challenge to a law or regulation as applied to the facts of a particular case.’” Kalin v. Nicholson, 172 Fed.Appx. 1000, 1002 (Fed.Cir. 2006). Thus, the findings of fact made by the BVA are critical to the success or failure of a veteran’s claim.

The BVA’s workload has increased dramatically in the past decade, reaching 85,288 decisions in fiscal year 2018. [6, p. 32] The vast majority of appeals (96%) considered by the BVA involve claims for compensation. [6, p. 31] Therefore, identifying the patterns of factual reasoning within the decisions of the BVA presents a significant challenge for automated argument mining.

For each of the 50 BVA decisions in our PTSD dataset, we extracted all sentences addressing the factual issues related to the claim for PTSD, or for a closely-related psychiatric disorder. This set of sentences (“PTSD-Sent”) is the dataset on which we conducted our experiments. The “Reasons and Bases” section of the decision is the longest section, containing the Board’s statement of the evidence, its evaluation of that evidence, and its findings of fact on the relevant legal issues. 3.1.1

For the purpose of identifying reasoning or argument patterns, we focus primarily on sentences that play one of three rhetorical roles in evidence assessment: the finding of fact, which states whether a propositional condition of a legal rule is determined to be true, false or undecided; the evidence in the legal record on which the findings rest, such as the testimony of a lay witness or a medical record; and the reasoning from the evidence to the findings of fact. Identifying the sentences that have those roles within adjudicatory decisions, however, presents special problems. Such decisions have a wide diversity of roles for sentences: e.g., stating the legal rules, policies and principles applicable to the decision, as well as providing citations to authority; stating the procedural history of the case, and the rulings on procedural issues; summarizing the evidence presented and the arguments of the parties based on that evidence; and stating and explaining the tribunal’s findings of fact based on that evidence. [ 37 ] Thus, BVA decisions pose the challenge of classifying rhetorically important types of sentence and distinguishing them from other types of sentence.

The following are the 5 rhetorical roles that we used to classify sentences in the PTSD-Sent dataset. Sentences were classified manually by teams of 2 trained law students, and they were curated by a law professor with expertise in legal reasoning. Data validity is open to scrutiny because the data will be publicly available.

Finding Sentence. A Finding Sentence is a sentence that primarily states a “finding of fact” – an authoritative conclusion of the trier of fact about whether a condition of a legal rule has been satisfied or not, given the evidence in the case. An example of a Finding Sentence is: “The most probative evidence fails to link the Veteran's claimed acquired psychiatric disorder, including PTSD, to active service or to his service-connected residuals of frostbite.” (BVA1340434)1

Evidence Sentence. An Evidence Sentence primarily states the content of the testimony of a witness, states the content of documents introduced into evidence, or describes other evidence. Evidence sentences provide part of the premises for findings of fact. An example of an Evidence Sentence is: “The examiner who conducted the February 2008 VA mental disorders examination opined that the Veteran clearly had a preexisting psychiatric disability when he entered service.” (BVA1303141)

Reasoning Sentence. A Reasoning Sentence primarily reports the trier of fact’s reasoning underlying the findings of fact (therefore, a premise). Such reasoning often involves an assessment of the credibility and probative value of the evidence. An example of a Reasoning Sentence is: “Also, the clinician’s etiological opinions are credible based on their internal consistency and her duty to provide truthful opinions.” (BVA1340434)

A unit of argument or reasoning within evidence assessment is usually composed of these three types of sentence (finding, evidence, and reasoning). The “Reasons and Bases” section of a BVA decision generally also includes two other types of sentence (those stating legal rules and citations), which must be distinguished from the first three. Unlike the case-specific elements of evidence, reasoning and findings, the legal rules and citations are often the same for tens of thousands of cases, even though the sentences stating those rules and citations can be highly variable linguistically, depending upon the writing style of the judge.

Legal-Rule Sentence. A Legal-Rule Sentence primarily states one or more legal rules in the abstract, without stating whether the conditions of the rule(s) are satisfied in the case being decided. An example of a Legal-Rule Sentence is: “Establishing direct service connection generally requires medical or, in certain circumstances, lay evidence of (1) a current disability; (2) an inservice incurrence or aggravation of a disease or injury; and (3) a nexus between the claimed in-service disease or injury and the present disability.” (BVA1340434)

Citation Sentence. A Citation Sentence references legal authorities or other materials, and usually contains standard notation that encodes useful information about the cited source. An example is: “See Dalton v. Nicholson, 21 Vet. App. 23, 38 (2007); Caluza v. Brown, 7 Vet. App. 498, 511 (1995), aff'd per curiam, 78 F.3d 604 (Fed. Cir. 1996).” (BVA1340434)

The frequencies of sentence rhetorical types within the PTSDSent dataset are shown in Table 1.

From the common dataset of 50 BVA decisions we randomly drew a set of 5 decisions to function as the qualitative-study observation sample (“QS-OS”). The QS-OS is the sample of labeled sentences that we studied qualitatively to hypothesize classification features for rhetorical roles. The QS-OS dataset contains 530 sentences, with the following frequencies for particular sentence roles: Finding Sentences = 58, Evidence Sentences = 201, Reasoning Sentences = 40, Legal-Rule Sentences = 81, Citation Sentences = 103, and other Sentences = 47. The veteran has a disability that is “service-connected”.

AND [1 of 3] The veteran has “a present disability”.

OR [1 of …] The veteran has “a present disability” of posttraumatic stress disorder (PTSD), supported by “medical evidence diagnosing the condition in accordance with [38 C.F.R.] § 4.125(a)”.

OR [2 of …] … AND [2 of 3] The veteran incurred “a particular injury or disease … coincident with service in the Armed Forces, or if preexisting such service, [it] was aggravated therein”.

OR [1 of …] The veteran’s disability claim is for service connection of posttraumatic stress disorder (PTSD), and there is “credible supporting evidence that the claimed in-service stressor occurred”.

OR [2 of …] … AND [3 of 3] There is “a causal relationship [“nexus”] between the present disability and the disease or injury incurred or aggravated during service”.

OR [1 of …] The veteran’s disability claim is for service connection of posttraumatic stress disorder (PTSD), and there is “a link, established by medical evidence, between current symptoms and an inservice stressor”.

OR [2 of …] …

Theorists on sample size for qualitative studies have determined that the appropriate size depends upon many factors [ 24 ]. They recommend that researchers can stop adding to the observation sample once that sample has reached reasonable “saturation,” such that it is sufficiently information-rich and adding more members would be redundant. [ 24, 12 ] In the present study, rather than devising a metric for saturation, we decided to test our main hypothesis by randomly drawing a very small sample of 5 decisions, analyzing the 58 sentences labeled as Finding Sentences in those decisions, forming hypotheses about predictive classification features, and testing the predictive power of those features.

For our ML experiments, we started with the entire PTSD-Sent dataset and performed certain preprocessing. We removed sentences that are merely headings, as well as numeric strings in the data. All words that remained were stemmed using NLTKs Snowball stemmer. Since punctuation symbols such as hyphens appear to interfere with the stemmer, we filtered out all nonalphabetic characters prior to the stemming step. If the filtering and stemming processes reduced a sentence to only blank characters, the entire sentence was dropped. Importantly, English stop words were not eliminated. Considering that each instance is a relatively short English sentence, eliminating any words might increase the classification error rate.

This preprocessing stage reduced the total data set to 5,797 usable labeled sentences. The frequencies of sentence types after preprocessing were: Finding Sentences = 490, Evidence Sentences = 2,419, Reasoning Sentences = 710, Legal-Rule Sentences = 938, Citation Sentences = 899, and other Sentences = 341.

The features chosen for the machine learning algorithm were the individual tokens in all the sentences (3,476), and the bigrams (30,959) and trigrams (59,373) that appear in them. These features also form the vocabulary for the vectorizer. We used the CountVectorizer class of the Scikit-learn Machine Learning library [ 25 ] as the feature extractor. The size of the vector was equal to the vocabulary size (93,808). On average, each sentence had about 60 true entries. 4

This Section describes the experiments we conducted in the qualitative study, as well as the results of those experiments. As we discussed in Section 3.2, the qualitative study was designed to test our main hypothesis that we can use a very small observational sample (only 5 decisions, containing 530 labeled sentences) to develop classifying scripts that perform reasonably well against the remainder of the PTSD dataset (a test dataset of 45 decisions, containing 5,422 labeled sentences), at least for purposes of some use cases. We also use the qualitative study to test our second hypothesis that attribution theory provides a general and transferable method for creating semantic types and linguistic features.

In order to develop a systematic methodology for discovering linguistic features that might classify Finding Sentences, we used attribution theory. An example of a sentence explicitly stating an attribution relation is: The Board finds that the veteran currently has PTSD. In interpreting the meaning of this sentence, we attribute to “the Board” the conclusion that “the veteran currently has PTSD”. As illustrated in this example, attribution relations have at least three elements or predicate arguments [ 22, 41 ]: (A) the attribution cue that signals an attribution, and which provides the lexical grounds for making the attribution (in the example, finds that); (B) the attribution subject, or the actor to which we attribute the propositional content of the sentence (in the example, the

Board); and (C) the attribution object, or the propositional content that we attribute to the attribution subject, expressed in normal form by an embedded clause (in the example, the veteran currently has PTSD). We distinguish the attribution cues and attribution subjects, on the one hand, from the proposition being attributed. We call the former “finding-attribution cues” because a lawyer uses them to determine whether a sentence states a finding of fact or not, regardless of which legal-rule condition might be at issue. The proposition being attributed, on the other hand, is the content of the finding. In the example above, the finding-attribution cues are “The Board finds that”, while the attribution object is the proposition “the veteran currently has PTSD.” An important reason for separating these two categories and testing their performance independently is that finding-attribution cues are more likely to be transferable to disabilities other than PTSD, and they are more likely to have counterparts even in other areas of law.

We conducted a qualitative study of the finding-attribution cues that occur within QS-OS, and ran various experiments to determine how scripts built on those cues would perform against QS-TSR&B. This section reports the results of several of those experiments, with the results tabulated in Table 2. 4.2.1

It appeared from the QS-OS that a highly-predictive single word might be “finds”. Although in this experiment we did not perform part-of-speech tagging, the word “finds” is generally used as a main verb (present tense, singular) when the Board states a finding. This is contrasted with Evidence Sentences, in which the verb is generally in the past tense (e.g., “found”), and the sentence attributes a proposition to a witness or document in the evidentiary record. We also observed occurrences of “concludes” and “grants” used in the same way as “finds”. We ran these three alternatives as a single experiment, using the regular expression (finds | concludes | grants), with the results shown as E1 in Table 2.

As shown in Table 2, a common mis-classification in E1 was with Legal-Rule Sentences. In Section 4.3 below, we discuss why precision is important for our use cases. By examining the LegalRule Sentences in QS-OS, we noted that, consistent with our main hypothesis, certain types of words and phrases occur in those sentences that we use to attribute them to legal authorities as sources of general legal rules. Such words and phrases include indefinite noun phrases (such as “a veteran,” as contrasted with “the Veteran”), conditional terms (such as “if” and “when”), and words typically used as cues for attributing propositions to higher courts (such as “held that” or “ruled that”). We tested scripts that used such words or phrases to exclude Legal-Rule Sentences from the results of E1, with the results shown in Table 2 for E1N. 4.2.2

A primary strength of a qualitative study is being able to identify a phrase that might be highly predictive of Finding Sentences due to the legal meaning of the phrase. One such phrase is “preponderance of the evidence”, which is used to formulate the legal standard for finding a proposition to be a fact. An alternative phrase that is often used when assessing what the total evidence proves is “weight of the evidence”. We ran scripts using these two alternatives against QS-TS-R&B, with the results shown in Table 2 as E2.

In order to test a combination of scripts, we ran a script that classified a sentence as a Finding Sentence if either E1 so classified it or E2 did so. The results are shown as E1+2 in Table 2. We also ran a combined experiment, including the Legal-Rule Sentence exclusion scripts from E1 (E1N) and from E2 (E2N), with the results shown as E1N+2N in Table 2.

We emphasize that we had a very limited objective in these experiments: to test, in a preliminary way, whether we could use attribution theory to develop hand-crafted, rule-based scripts that could perform adequately in a variety of important use cases. If we could observe useful linguistic patterns in only 5 decisions, we might be able to develop a general methodology that would be transferable to adjudicatory decisions in many areas of law.

We also stress that whether performance is adequate is a function of the end use case. For example, if the use case is to retrieve similar cases and to highlight sentences by rhetorical type for the purpose of suggesting how similar evidence has been argued in past cases, then the priority might be on precision over recall. This is because wasting the user’s time with non-responsive returns might have a more serious cost than merely failing to retrieve all similar cases. For such a use case, even recall = 0.436 (for E1N+2N, Table 2) might be useful because nearly half of all Finding Sentences were correctly identified (true positives).

In addition, precision = 0.647 (for E1N+2N, Table 2) might be acceptable, because the false positives (sentences incorrectly classified as Finding Sentences) constituted only about 1/3 of the predicted sentences. Moreover, the largest number of mis-classified sentences occurred in Reasoning Sentences (68). This may be because a judge might use a main verb such as “finds” when reporting the Board’s intermediate reasoning about the credibility or persuasiveness of individual items of evidence. Of the incorrectly classified sentences, about 80% were Reasoning Sentences, which are probably also instructive to a user who is looking for examples of arguments about evidence. For such a use case, a user might learn as much or more from reviewing a Reasoning Sentence as from reviewing a Finding Sentence, and confusion between these two rhetorical roles is less important. For these use cases (semantic search and semantic viewer), the performance of even these simple scripts could be very useful.

Contrast such use cases with a use case that calculates a probability of success for an argument pattern, based on historic results in decided cases. For such a use case, the validity of the probability would depend critically upon relative frequency in the database, and on high recall and precision of similar arguments from past cases. Retrieving every similar case would be a priority with a potentially significant cost of error – e.g., reliance on an erroneous probability in deciding whether to bring or settle a new legal case. Moreover, confusion between Finding Sentences (which record whether an argument was successful or not) and any other rhetorical type of sentence could have significant consequences.

Because we based the script development for these experiments on attribution theory, as well as on general concepts used to increase precision, we expect this methodology to be transferable to other legal areas besides veterans’ disability claims. 5

This Section describes the experiments we conducted in the ML study, as well as the results of those experiments. As described in Section 3.3, we filtered out certain sentences from the dataset, and stemmed the words, leaving us with a preprocessed dataset of 5,797 labeled sentences. Our goal was two-fold: to assess how well the chosen machine learning classifiers perform relative to each other and to the qualitative-study scripts; and to find out which features were determined by each classifier as being significant to the prediction of Finding Sentences. The algorithms we chose for this study are Naive Bayes (NB), Logistic Regression (LR), and support vector machines (SVM) with a linear kernel [ 26, 9, 8 ].

We ran each ML algorithm 10 times, each run using a randomly chosen training subset that contained 90% of the labeled sentences. The trained classifier was used to predict the labels for the remaining 10% of sentences. All results shown in this section are the averages over these 10 runs.

For each ML algorithm, we ran two sets of experiments. In the first set of experiments (the “multi-class” experiments) we retained the labels for all 5 sentence types in the PTSD-Sent dataset – i.e., each classifier was fit to a multi-class training set. We recorded the overall accuracy score (the fraction of correctly labeled test instances), the classification summary, and the confusion matrix for each algorithm and each run. The classification summary records the precision, recall and F1-score for each label. A confusion matrix cell-value C[i][j] is the number of test sentences that are known to be in class i (row i) but are predicted by the classifier to be in class j (column j). All values shown are averages over the 10 runs.

In the second set of experiments (the “two-class” experiment), we labeled all sentences other than Finding Sentences as “NonFinding” sentences, so the training and test datasets then contained only two classes. As before, we recorded the accuracy scores, the classification summaries, and the confusion matrices and averaged them over the runs of each algorithm. In addition, for the LR and SVM classifiers, we extracted the top 20 features as measured by their weights in the fitted classifier. Note that since Finding Sentences form only about 8.5% of the dataset, the default classifier that labels all test instances as “Non-Finding” would have an accuracy score of 91.5% (under reasonable assumptions about the distribution of sentences).

Table 4 summarizes the average accuracy for each classifier, for each of the two sets of experiments. We also computed the false positive rates from the confusion matrices. The remainder of this section of the paper reports details on each ML classifier.

The Scikit-learn Python module has implementations of multiple variants of the basic NB algorithm. We chose the GaussianNB implementation with default parameters to present results (implementation of ComplementNB yielded similar results). Results for the two-class experiment are shown in Table 5.

The LR algorithm produces a binary classifier, also known as a loglinear classifier. Since the LR algorithm produces only binary classifiers, for our multi-class experiments we used the one-versusthe-rest approach. Results are shown in Tables 6 – 8.

Discussion: The results show that LR is an acceptable classifier for this problem. The two-class accuracy score of 96.3% (Table 4) is better than that of the default classifier, although in this classifier as well most of the accuracy score appears to come from the correct predictions of the Non-Finding Sentences. The two-class precision of 0.84 for Finding Sentences (Table 8) indicates that false positives are still a concern, though substantially lower than those of the NB classifier. The confusion matrix did not indicate any dominant source of error. The words and phrases (stemmed) in the highest-ranked features were similar to those used in the handscripted classifier.

An SVM is an ML algorithm for binary classification problems. It is based on finding a maximum margin hyperplane that divides the training set into the two classes. Based on the success of the LR classifier, we decided to use a linear kernel for the SVM. Since SVM classifiers are by default binary, for the multi-class experiment the implementation builds one-versus-one classifiers and a voting scheme is used to predict the label for a test instance. Some results are shown in Tables 9 and 10.

Discussion: The results show that performance of the SVM classifier with a linear kernel has similar performance to that of the LR classifier. This is true for both the multi-class and the two-class experiments. However, there is substantial divergence in the top features chosen by the two algorithms. The features in common are “board find”, “thus” and “whether”. One hypothesis is that most of the top features are used to decide the Non-Finding class labels, and the Finding class arises as a default class. Several of the highestranked features seemed to be specific for PTSD cases. Also, as with the LR classifier, the confusion matrix for the multi-class SVM did not indicate any dominant source of classification error.

We used attribution theory to develop a qualitative methodology for analyzing a very small sample of labeled sentences to create rule-based scripts that can classify sentences that state findings of fact (“Finding Sentences”). We compared the results of those scripts with the performance of standard ML algorithms trained and tested on a larger dataset, but one that is still a relatively small dataset by ML standards. Both of these experiments suggest that some access-to-justice use cases can be adequately addressed with very small quantities of labeled data, and at much lower cost than previously believed.

We thank the Maurice A. Deane School of Law for its support for the Research Laboratory for Law, Logic and Technology.