This paper proposes a new computational architecture for narrative-driven case elicitation, describes six new legal narrative corpora, and evaluates two diferent approaches to creating legal narrative schemas, the first using language models, and the second using event sequence alignment. An experimental evaluation suggests that the sequence alignment approach may be more appropriate for legal corpora that are small, sparse, and heterogeneous.
instantiated narrative templates with user-provided facts,
none are able to interpret text provided by a user or
asIncreasing numbers of litigants worldwide face the chal- sist a user in paraphrasing facts in a manner likely to
lenges of representing themselves in courts and other communicate them most efectively to a judge.
decision forums without the assistance of an attorney. Legal aid attorneys often elicit case facts by starting
[1] [2]. Significant reductions in public legal-aid expendi- with a general question (e.g., “How can I help you
totures in many jurisdictions have fueled this trend, leading day?”) followed by a series of follow-up questions to
to increases in self-represented litigants (SRLs) in the UK ifll-in missing parts of the client’s story while ignoring
[3], the EU [4], Canada [5] [
Many technological innovations, such as Online Dis- narrative fields of petitions or other court documents.
pute Resolution [
Proceedings of the Sixth Workshop on Automated Semantic Analysis of Information in Legal Text (ASAIL 2023), June 23, 2023, Braga, Portugal * Corresponding author. " lbranting@mitre.org (K. Branting); smcleod@mitre.org (S. McLeod); blp73@cornell.edu (B. Park); karine@mitre.org (K. Megerdoomian) 0000-0002-9362-495X (K. Branting)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Section 4 overviews the process of converting text to CPWErooUrckReshdoinpgs ISNc1e6u1r-3w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) event sequences, and six corpora of legal narratives of representative types are described in Section 5. Section 6 indicate stereotypical sequences of events that create sets forth two approaches to narrative schemata induc- expectations and fill in missing details [19]. tion—one based on language models and a second based Induction of narrative schemas (i.e., scripts in on sequence alignment—and presents experimental re- Schank/Abelson terminology) was pioneered by Chamsults in predicting missing narrative events. The implica- bers and Jurafsky [20] who defined ”narrative chains” as tions of these results and proposals for future work are “partially ordered set[s] of narrative events that share a presented in the final section. common actor” and use pointwise mutual information (PMI) as a measure of event association strength. Subsequent work showed that using argument consistency 2. Related Work as a criterion for event relatedness improved model predictiveness as measured by a narrative cloze test, i.e., Our research in creating and using legal narrative predicting a missing event [21]. However, even when schemas for fact elicitations connects two complemen- trained on the Gigaword Corpus, performance was surtary strands of prior work: investigations of the role of prisingly weak, with the average “ranked position” of story understanding legal client interviewing; and induc- over 1,050 under the best performing condition. tion of narrative schemas to support automated story Subsequent work introduced skip grams to compenunderstanding. sate for data sparseness, language modeling formalisms better suited to cloze prediction (e.g., bigram probability 2.1. The role of story understanding in rather than PMI), and recall@n rather than average rank case elicitation as an evaluation metric [22]. A separate approach applied multiple sequence alignment to event sequences then extracting and simplifying the graph formed by treating each row as a node and adding edges to pairs of nodes that contain events that were consecutive in some event sequence [23].
Improved narrative cloze performance results were obtained by stricter constraints on multi-argument consistency [24], topic-specific training sets [ 25], and alternative language models, e.g., Hidden-Markov [26], LogBilinear [27], and Association Rule models [28]. However, no significant eforts appear to have been directed to the task of induction of legal narrative schemas or the use of such schemas in fact elicitation.
The facts of legal cases are more than mere collections of
events. Rather, case facts are narratives having settings,
characters with goals and motives, and events linked by
temporal, causal, and intentional relations. Outcomes of
trials often depend on the relative story-telling ability
of attorneys and witnesses [14], and jurors have been
shown empirically to decide cases based on which of
two competing narratives imposes the highest degree of
coherence on the evidence presented at trial [15] [
When interviewing a client to determine the client’s story, attorneys often try to elicit “the causal and temporal connections that contribute to giving the events contextual meanings . . . with the aim of defining ‘Who has done what, how, when, why and where?’” [17]. Clients’ narratives are often “redefined to be a legally relevant narrative” by legal aid attorneys, a process that can sometimes interfere with or prevent understanding of emotionally salient background information if it is too rigid [18].
Notwithstanding the central role of narrative in legal client interviews, there has been little exploration of techniques for automating narrative elicitation. A paucity of operational theories of text-based narrative analysis may have played a role in the lack of activity in this area.
Recent advances in narrative schema induction have enabled the novel research described below in this paper on narrative case elicitation.
perform two functions: acquiring narrative schemas from examples; and using those schemas to guide interactions with litigants. Figure 1 sets forth an architecture that performs these two functions.
The left side of Figure 1 details an of-line mechanism for inducing schemas from narrative corpora, which is the primary focus of this paper. The right side of Figure 1 second depicts a real time component that uses these schemas to distinguish relevant from irrelevant utter2.2. Narrative Schema Induction ances and to identify facts that could distinguish among The importance of narrative schemas for story under- legal schemas if confirmed or disconfirmed. Specifically, standing was recognized early in the history of AI. Roger each litigant’s utterance is converted into a sequence of Schank and Robert Abelson coined the term “scripts” to events to be added to the event sequence derived from prior utterances. The combined sequence is then compared to one or more narrative schemas. This comparison among events by resolving coreferences and determining permits relevant events (those that match) to be distin- the discourse relations among the events. Many alterguished from irrelevant events (unmatched events) and native approaches could be used to perform these two can suggest missing events (unmatched schema events) steps; we use ANAnSI (Advanced Narrative Analytics that should be inquired about. The Text Realizer gener- System Infrastructure) [30], a system that integrates the ates questions to determine whether missing events can output of the Stanford Core NLP [31] constituency parser be confirmed or disconfirmed. Additional events elicited and cTakes [32] into a temporal, causal, and intentional in this manner can distinguish among partially match- graph represented in Neo4j [33] (see Figure 2). ing narratives or refine the match to the most similar narrative. 4.1. Graph Linearization
The real-time processing depicted on the right side of Figure 1 depends on the existence of a narrative schema The resulting graph representation for a collection of one for each area of law for which facts are to be elicited.1 The or more sentences is then linearized into an event seprocess of induction of schemata from event sequences, quence with arguments and semantic roles standardized depicted on the left side of Figure 1, is detailed in the in the manner proposed in [24] to three alternative roles: next section. However, both the ofline and real-time agent, patient, and other complement. For example, in aspects of depend on conversion of raw text into event the event sequence shown in Figure 3, the pronouns “I” sequences, as shown as the second and third steps on and “me” are normalized to “I”. both sides of Figure 1.
We term this architecture RIM, short for “Relevant, 4.2. Lemma Normalization Irrelevant, and Missing,” since the key functionality of the system is identifying these three categories of events.
parse each sentence into individual events and, for each event, identify the entities that fill the semantic roles of that event. The next step is analyzing the relationships
As discussed below in Section 5, corpora of legal narratives are, in general, many orders of magnitude smaller than the corpora used in previous narrative schema elicitation research, such as the Gigaword corpus [34]. Such small corpora produce sparse transition matrices with little predictive value, e.g., most event pairs in a new (or held out) event sequence will have never been seen before, meaning that there is no frequency data on which to base cloze predictions.
Several normalizations were therefore applied to reduce vocabulary size to improve matching. The most important and general of these was lemma normalization, which consists of clustering events in semantic embedding space2 and replacing each event with the most central member of the cluster in which it occurs. For example, Figure 4 show the results of complete-linkage hierarchical clustering of events in the EEOC (Equal Employment Opportunity Commission)3 corpus (described below) with a minimum cosine threshold of 0.75.4 For example, both “harass” and “threaten” are replaced by “intimidate,” and ‘ask,” “hear,” “know,” and “let” are all replaced by “tell.’ ’
A second normalization that was motivated by the EEOC domain but useful in other domains was to replace each occurrence of a form of “to be" that has as an argument the name of an occupation with the event “be OCCUPATION.”5 2We used the spaCy large English model, https://spacy.io/models/en. 3See https://www.eeoc.gov/. 4This threshold is an ad hoc setting, intended to be low enough to group synonymous terms without merging terms with obviously diferent meanings. 5We used the list of 1,156 occupations, from “accountant" to “zoologist” set forth in https://github.com/johnlsheridan/occupations/ blob/master/occupations.csv. We used the occupational normaliza
the narrative, increasing transition matrix density and therefore increasing the likelihood that event cooccurrences will have been observed in the training corpus. This reduction in vocabulary size comes at the cost of reducing the specificity of the event representation.
A key challenge for narrative schema-based case elicitations is the dificulty of obtaining significant numbers of narrative texts representative of narratives produced by litigants. In general, such texts contain sensitive personal information that precludes sharing in the form of public corpora. Documents filed in legal or administrative bodies are typically public, so statements of facts in petitions, complaints, and other filings can be a source of narrative texts. However, counsel for litigants often draft the statements in facts of court filings, so the text of such statements seldom contains language used by litigants tion in all experiments below. themselves except in the case of self-represented (pro se) legal narratives to guide interactions with a litigant based litigants, i.e., those who have no attorney to draft their on distinguishing relevant from irrelevant facts as they statements in fact. The ideal corpus would consist of are presented and predicting missing facts that would statements of fact in pro se litigants’ filings, but such contribute to a coherent story. The relative efectiveness iflings are dificult to obtain in bulk. of narrative models for each of these activities can be
In this research, we obtained one small corpus of texts estimated using narrative cloze tests, which estimate the by pro se litigants together with five other data sets ability of models to predict a missing (typically, the next) intended to reflect various characteristics of fact state- event in a sequence. ments: We explored two types of predictive models: language models; and event sequence alignment models.
scribed from handwritten texts in the field titled “The facts supporting the plaintif’s claim of dis- 6.1. Language Models crimination,” in thirty employment discrimina- For each of the 6 data sets described above in Section 5 tion complaints filed in the Northern District of we converted each narrative text into a linearized event Illinois in 2016. These texts are representative of sequence, with events lemmatized by clustering in selitigant-generated narrative texts. mantic vector space with a similarity threshold of 0.75. 2. Multi-LexSum Summaries of Civil Cases. These We calculated the recall@n in 10-fold cross validation. 364 summaries of civil rights lawsuits were cre- For these experiments, we relaxed the constraint that ated for training and evaluating legal case sum- cooccurring events share common arguments to reduce marization [35]. The Multi-LexSum text were the efects of data sparsity. included to typify procedural histories, a type of Several aspects of the results, shown in Table 2, suggest narrative required for appeals that court person- that data sparsity in narrative corpora of the magnitude nel have identified as being challenging for pro of those evaluated in this experiment present a signifse appellants. icant impediment to their use in the RIM framework. 3. WIPO cases. The “background facts” of 6,000 deci- First, little improvement was observed between unigram sions by World Intellectual Property Organization and trigram models, suggesting that there are too few in domain name disputes. These fact statements multi-event sequences for efective training. Moreover, were drafted by the panel deciding the case and there was only a modest improvement from recall@1 are therefore not representative of pro se text. to recall@10, suggesting that many transitions in test However, the similarity among these fact state- data were never observed in the training data. Thus, data ments suggests that they could be a benchmark sparsity appears to remain a significant issue even after for narrative induction. reducing the vocabulary size through semantic cluster4. Board of Veteran Afairs decisions. The “Introduc- ing.
tion” section of 1,680 Board of Veterans Appeals (BVA) cases. As with the WIPO cases, these texts 6.2. Sequence Alignment Models are drafted by the judge writing the opinion and are therefore not representative of pro se text but An alternative approach to narrative schema induction potentially useful as a benchmark for narrative that may be more appropriate for domains with very induction. sparse training data is based on event sequence align5. SPOT-HO online housing questions. Two hun- ment. In this approach, which is inspired by techniques dred sixty three questions posed to the Sufolk of molecular biology, event sequences are aligned to find University Law School’s Legal Innovation and the most common subsequences, which can then be used Technology (LIT) Lab issue spotting service [36]. as components of narrative schemas. Figure 5 shows the 6. SPOT-WO online employment questions. Two local alignment, that is, the alignment maximizing the hundred ninety five employment questions posed longest common subsequence (LCS)[37], between two to the SPOT site. event sequences from the BVA corpus. Normalized LCS (NLCS) as shown in Formula 1 is a metric useful for comparing and grouping similar event sequences, even if they difer in lengths.
1.0 − (|(1, 2)|/(|1|, |2|))) (1)
SPOT-WO
SPOT-HO Multi-Lexum
BVA WIPO Size 30 295 263 364 1,680 6,000 complaints legal advice requests legal advice requests procedural history background facts background facts pro se litigant lay public lay public federal judge administrative law judge administrative law judge
common subsequences. 2. Merge. Create individual models from each group of sequences. 3. Match. Use each model to distinguish relevant, irrelevant, and missing events from new sequences.
that makes them useful for recognizing new event sub- 6.2.1. Alignment-Based Sequence Clustering sequences, e.g., that might share diferent subsequences with diferent cluster members. Consistent with this intuition, we perform the following steps to convert each corpus to a model consisting of a set of schema: Multiple sequence alignment is quite computationally expensive for collections of sequences in the size range of the 6 corpora in our experiments (30-6,000), so we use a heuristic approach derived from the center star alignment algorithm of [39].
1. Convert narratives to event sequences, as per Sec
tion 4. 2. Perform total-linkage agglomerative hierarchical clustering with distance metric NLCS and distance threshold . The resulting clusters comprise event sequences that share a significant proportion of event subsequences.
event and each cooccurring pairs of events is connected by an edge from the earlier to the later event. 3. For each non-medoid sequence, , in the cluster, align and merge with the DAG by combining each node of with corresponding node in the best matching path in . If is unmatched, it is added as a new node in the DAG. 6.2.2. Merging Event Sequences
For each cluster, , of similar event sequences, we perform the following steps to merge the sequences into a schema: 1. Identify the medoid, i.e., the sequence having the highest mean similarity to the other members of the cluster, breaking ties in favor of shorter sequences. 2. Convert the medoid sequence into a directed acyclic graph (DAG) in which each node is an 6.2.3. DAG Matching In our initial procedure for matching a new sequence with a DAG, we identified the alignment between and each unique path in the DAG to find the path that maximizes the matched portion of . We hypothesized that we would typically obtain a better match from the DAG than from any one of the individual events sequences merged into that DAG. (the minimum number to compare the DAG matching with matching to individual cluster members). The third row, “compression,” represents the number of nodes in the DAG divided by the number of events in the event sequences composing that DAG, e.g., the proportion of overlap among the event sequences. The remaining rows show the precision, recall, and f-measure under the control condition (the model consisted simply of whatever case event sequence had the highest LCS) and the test condition (the model was the best DAG path with the highest LCS). 6.2.5. Results 0.85 0.44 0.87 0.331 0.042 0.079 0.343 0.024 0.046
in the research program of narrative-guided fact elicitation for self-represented litigants. Acquisition by the research community of larger datasets of legal narratives, particularly those produced by self-represented litigants, is a vital next step for progress in this important problem.
der for permitting us to experiment with her collection of EEOC complaints and to Charles Horowitz for assistance with ANAnSI. The MITRE Corporation is a not-for-profit company, chartered in the public interest. This document is approved for Public Release; Distribution Unlimited. Case Number 23-1184. ©2023 The MITRE Corporation. All rights reserved.