This paper proposes an approach for narrative-driven case elicitation that uses schemas induced from corpora of legal case facts to distinguish relevant from irrelevant utterances and to identify facts that could distinguish between competing hypotheses. This approach to narrative-driven case elicitation builds on recent research in narrative schema induction.
Law is full of stories, whether these are
stories that are told in the courtroom as
lawyers try to weave together compelling
and competing versions of an event, in the
legislative histories that subtend a body
of statutes, or in stories about the origins
and acceptance of legal systems [
of events. Instead, case facts are narratives that have settings, characters with goals and motives, and events triggered by the characters’ actions. Just as not every set of facts is a story, not every story is legally significant.
It is the role of a legal counsel to identify the goals that the client hopes to achieve through a legal process and 2. The Role of Narrative in Law then to elicit a coherent narrative that is relevant to one or more possible legal remedies that could achieve those Law can be viewed as a framework of rules under which goals. legal arguments often consist of alternative narratives
Lay (non-attorney) clients typically have little under- that lead to opposite consequences. Empirical evidence
standing of what facts are relevant to a possible legal has shown that jurors often decide cases based on which
remedy. Accordingly, attorneys must help clients express of two competing narratives imposes the highest degree
the facts that are relevant to possible legal remedies and of coherence on the evidence presented at trial [
This paper sets forth an approach to narrative-driven client interviews revealed the importance of permitting
case fact elicitation and situates that approach within a a client ample time to speak, during which the attorney
acknowledges the information received and expresses
interest but otherwise “refrains from interrupting”
except to direct the narrative away from “precarious” legal
grounds and toward “possible remedies” “until the client
has gone on for long enough to establish the problem at
hand” [
In the context of protective order interviews, it has sured by a narrative cloze test, i.e., accuracy in
predictbeen shown that the interviewer “acts to reshape, if not ing the next or a missing event [16]. Improving
narrepair, the narratives of domestic violence victims, so rative cloze performances has been obtained by using
that they conform to the requirements of an afidavit stricter constraints on multi-argument consistency [17],
that must be submitted to a judge if a protective order is topic-specific training sets [ 18], and alternative language
to be issued” [
In summary, the achievement of a client’s goals de- standing the limitation that narrative schemas are, as pends on the degree to which the clients story can be mentioned above, only a partial representation of the elepresented, whether in a written petition or to a judge or ments of story. This is because the aspect of performance jury, in a compelling, coherent, and legally relevant way. that narrative schema induction seeks to optimize, the This depends in turn on the ability of the client’s (human cloze test, can play a central role in the model of legal or automated) counsel to elicit case facts in a manner case elicitation, as described in the next section. that has these characteristics.
A story has been described as “a character-based and Section 2 described how efective case elicitation requires
descriptive telling of a character’s eforts, over time, to helping a client articulate the events giving rise to a
overcome obstacles and achieve a goal” [
Unlike story grammars, scripts ordinarily lack the hi- mation that is most relevant to a litigant’s legal goals, in erarchical and recursive structure needed to account for the sense of being the fact that would best discriminate some important properties of stories (e.g., in a picaresque among legally relevant narratives, is set forth in Algonovel or a protective order application there can be a rithm 1. The key requirement for the narrative elicitation variable number of episodes, and episodes can have sub- process is a set of <schema, goal> pairs, (SGs), where episodes). Moreover, both story grammars and scripts each schema is a model capable of evaluating the relative initially were entirely manually constructed, which made likelihood of a given event sequence, and each goal is a them unscalable. (possibly negated) legal remedy, e.g., a protective order, a
However, a series of research advances have made it child custody order, a finding of employment discriminaincreasingly feasible to induce “narrative schemas” (i.e., tion, etc. Each schema is induced from a corpus of related “scripts” in Schank/Abelson terminology) from examples. case facts for a given area of law using the techniques The seminal work by Chambers and Jurafsky [15] defined described in Section 3.1 “narrative chains” (later termed “narrative schemas” [16]) as “partially ordered set[s] of narrative events that share
such schemas from representative legal case facts [22].
objective that the client hopes to achieve through a legal process, the hypothesis is the best matching SG, and the facts consist of the events and relations elicited thus far from the client.
Algorithm 1 Legal narrative elicitation sgs ← {SG* } hypothesis ← ∅ facts← ∅ goal ← ask(goal) hypothesis ← BestMatch(facts,goal,sgs) while match(facts,goal,hypothesis)< threshold do newFact ← ask(getMissing(facts,hypothesis) add(facts,newFact) hypothesis ← BestMatch(facts,goal,sgs) end while return facts, goal, hypothesis
Algorithm 1 is only a high-level depiction of the actual complex process of case elicitation. The identification of the goal or goals of a client can itself be a complex process requiring mixed-initiative dialogue techniques beyond the scope of this paper. However, but there is an extensive body of research on goal-directed dialog (e.g., [23] [24]), and recent work has addressed the specific task of categorizing a legal aid clients’ problems [25].
The function “match(facts,goal,SG)” measures the degree of match between the client’s goal and current facts and the SG. An appropriate baseline function for “match” is the probability of the facts under the schema, or 0.0 if the goals don’t match (i.e., the schema is irrelevant if the goals don’t match, regardless of how well the facts match). The hypothesis is the currently best matching SG ∈ sgs. The function “getMissing(facts,hypothesis)” returns the missing fact or event that, if added to the facts, would most increase the probability of those facts.
Performance on this task is equivalent to performance on the narrative cloze test [16], so as advances in narrative schema induction improve performance on cloze, the “getMissing” function should improve as well. The hypothesis is updated after each new fact has been elicited, reflecting the way that an attorney’s assessment of a case may change as more facts are learned.
The process of eliciting additional facts and refining the hypothesis continues until there is no more progress or the match between the hypothesis and the facts and goal exceeds a success threshold. Only at this point are the legal rules for the remedy applied to the case facts.
This corresponds to an attorney eliciting the full story from a client before turning to an assessment of the viability of a claim based on that story.
Figure 1 illustrates the initial steps of an elicitation under process shown in Algorithm 1. The dialogue manager starts by asking the client’s goal (Step 1). The client replies by stating there was a threat and that the client wishes for help against the threat (Step 2). The dialogue manager uses this goal to instantiate a new case with a goal (Step 3) and the start of an event sequence. The matcher searches for the SG that best matches the goal and initial event sequence of the new case (i.e., the SG whose remedy matches the goal and whose schema maximizes the probability of the case’s fact sequence) and makes that SG the current hypothesis (Step 4).
Figure 2 shows the predictor using the schema of the current hypothesis to predict the most probable missing fact in the current case (Step 5) and the dialogue manager converting the fact into an appropriate discourse action (Step 6).
A more detailed (although still simplified) visualization A system for narrative schema-based fact elicitation deof the initial stages in Algorithm 1 is shown in Figure 3. pends on two coordinated capabilities: acquiring narraThe sequence of events starting with “meeting,” “inti- tive schemas from examples; and using those schemas to mate relationship,” etc., represents the most probable guide interactions with litigants. Figure 4 sets forth an event sequence under a typical protective order schema. architecture for providing these two capabilities. This arThe “case facts” represent a relational representation of chitecture is termed RIM, short for “Relevant, Irrelevant, a client’s answer to the question, "How did you meet?" and Missing,” since the key functionality of the system is
Any practical implementation of Algorithm 1 must identifying these three categories of events. ensure a common representation for the schemas and The left side of Figure 4 details an of-line mechanism case facts; otherwise, the matching and prediction steps for inducing schemas from narrative corpora. The right would not be possible. The next section describes an side of Figure 4 depicts the real time component, includarchitecture to achieve this common representation. ing the process described in the previous section of using these schemas to distinguish relevant from irrelevant utterances and to identify facts that could distinguish among legal schemas if confirmed or disconfirmed. The
Text Realizer generates questions to determine whether System Infrastructure) [26], a system that integrates the missing events can be confirmed or disconfirmed. Ad- output of the Stanford Core NLP [27] constituency parser ditional events elicited in this manner can distinguish and cTakes [28] into a temporal, causal, and intentional among partially matching narratives or refine the match graph represented in Neo4j [29]. to the most similar narrative. Figure 5 shows a portion of the graph for the sentence
The real-time processing depicted on the right side of “During my employment, Respondent placed me on a Figure 4 depends on the existence of a narrative schema leave of absence and required me to pass a medical exam for each area of law for which facts are to be elicited. The in order to return to work” showing temporal and intenprocess of induction of schemata from event sequences, tional links between pairs of events. depicted on the left side of Figure 4, is detailed in [22].
However, both the ofline and real-time aspects of depend 6.2. Graph Linearization on conversion of raw text into event sequences, as shown as the second and third steps on both sides of Figure 4.
The first step in converting text to event sequences is to parse each sentence into individual events and, for each event, identify the entities that fill the semantic roles of that event. The next step is analyzing the relationships among events by resolving coreferences and determining the discourse relations among the events. Many alternative approaches could be used to perform these two steps; we use ANAnSI (Advanced Narrative Analytics The resulting graph representation for a collection of one or more sentences is then linearized into an event sequence with arguments and semantic roles standardized in the manner proposed in [17] to three alternatives: agent, patient, and other complement. For example, in the event sequence shown in Figure 6, the pronouns “I” and “me” are normalized to “I”.
As discussed below in Section 7, corpora of legal narratives are, in general, many orders of magnitude smaller than the corpora used in previous narrative schema elicitation research, such as the Gigaword corpus. Such small corpora produce sparse transition matrices with little predictive value, e.g., most event pairs in a new (or held out) event sequence will have never been seen before, meaning that there is no frequency data on which to base cloze predictions.
Several normalizations were therefore applied applied 2We used the spaCy large English model, https://spacy.io/models/en. to reduce vocabulary size to improve matching. The most of a form of “to be” that has as an argument the name of important and general of these was lemma normaliza- an occupation with the event “be OCCUPATION.”3 tion, which consists of clustering events in semantic em- Lemma normalization shrinks the vocabulary size of bedding space2 and replacing each event with the most the narrative, increasing transition matrix density and central member of the cluster in which it occurs. For therefore reducing the likelihood that event cooccurexample, Figure 7 show the results of complete-linkage rences will never have been observed in the training hierarchical clustering of events in our EEOC corpus (de- corpus. This reduction in vocabulary size comes at the scribed below) with a minimum cosine threshold of 0.75. cost of reducing the specificity of the event representaFor example, both ‘harass’ and ‘threaten’ are replaced tion. by ‘intimidate,’ and ‘ask,’ ‘hear,’ ‘know,’ and ‘let’ are all replaced by ‘tell.’
A second normalization that was particularly useful in the employment domains was to replace each occurrence
ogist” set forth in https://github.com/johnlsheridan/occupations/ blob/master/occupations.csv.
A key challenge for narrative schema-based case elicitations is the dificulty of obtaining significant numbers of narrative texts representative of text produced by litigants. In general, such text contains sensitive personal information that precludes sharing in the form of public corpora. Documents filed in legal or administrative bodies are typically public, so statements of facts in petitions, complaints, and other filings can be a source of narrative texts. However, counsel for litigants often draft the statements in facts of court filings, so the text of such statements seldom contains language used by litigants themselves except in the case of self-represented (pro se) litigants, i.e., those who have no attorney to draft their statements in fact. The ideal corpus would consist of statements of fact in pro se litigants’ filings, but such iflings are dificult to obtain in bulk.
In this research, we obtained one small corpus of texts by pro se litigants together with five other data sets intended to reflect various characteristics of fact statements:
scribed from handwritten texts in the field titled “The facts supporting the plaintif’s claim of discrimination,” in thirty employment discrimination complaints filed in the Northern District of Illinois in 2016. These texts are representative of litigant-generated narrative texts. 2. Multi-LexSum Summaries of Civil Cases. These three hundred sixty four summaries of civil rights lawsuits were created for training and evaluating legal case summarization [30]. The Multi-LexSum text were included to typify procedural histories, a type of narrative required for appeals that court personnel have identified as being challenging for pro se appellants. 3. WIPO cases. The “background facts” of 6,000 decisions by World Intellectual Property Organization in domain name disputes. These fact statements were drafted by the panel deciding the case and are therefore not representative of pro se text. However, the similarity among these fact statements suggests that they could be a benchmark for narrative induction. 4. Board of Veteran Afairs decisions. The “Introduction” section of 1,680 Board of Veterans Appeals (BVA) cases. As with the WIPO cases, these texts are drafted by the judge writing the opinion and are therefore not representative of pro se text but potentially useful as a benchmark for narrative induction. 5. SPOT-HO online housing questions. Two hundred sixty three questions posed to the Sufolk University Law School’s Legal Innovation and Technology (LIT) Lab issue spotting service [31]. 6. SPOT-WO online employment questions. Two hundred ninety five employment questions posed to the SPOT site.
This paper has proposed an approach to narrative-driven case elicitation that builds on recent research in narrative schema induction. This approach uses schemas induced from corpora of legal case facts to distinguish relevant from irrelevant client utterances and to identify facts that could distinguish among competing hypotheses if confirmed or disconfirmed.
Only the ofline portion of the RIM model has been implemented in this project, as described in [22]. A working prototype of the narrative elicitation portion of RIM, which is the focus of this paper, would require assembling a library of scheme-goal pairs for a given area of law from a suitable corpus, such as the BVA or WIPO corpora described above. The matching and prediction functions described in Algorithm 1 are basic capabilities of the schemas described in [22], and as mentioned Corpus
EEOC SPOT-WO
SPOT-HO Multi-Lexum
BVA WIPO
Size complaints legal advice requests legal advice requests procedural history background facts background facts pro se litigant lay public lay public federal judge administrative law judge administrative law judge above there is an extensive literature on the goal-directed mixed-initiative dialogue techniques needed for the dialogue manager. Thus, there are no significant technical obstacles to implementing a prototype.
The narrative-driven case elicitation approach described here is quite unlike the dominant techniques for automated legal assistance, which are overwhelming organized around form-filling [ 32] or backchaining through logical representations of legal rules [33] (see generally [34]). The narrative-driven approach is intended to change the focus of client interviewing from the structure of the target legal artifacts (completed petitions and legal rules) to the life experiences that give rise to legal claims and to human interactions between client and attorney. While the narrative-driven model proposed here is a drastic simplification of the actual interview process between clients and attorneys, we hope that it will be a first step toward more faithful model and is feasible to implement with current narrative schema induction and dialogue technology.
der for permitting us to experiment with her collection of EEOC complaints and to Charles Horowitz and Karine Megerdoomian for their assistance with ANAnSI. The MITRE Corporation is a not-for-profit company, chartered in the public interest. This document is approved for Public Release; Distribution Unlimited. Case Number 23-1184. ©2023 The MITRE Corporation. All rights reserved. [14] R. C. Schank, R. Abelson, Scripts, Plans, Goals, S. Bethard, D. McClosky, The stanford corenlp natand Understanding, Hillsdale, NJ: Earlbaum Assoc, ural language processing toolkit., in: ACL (System 1977. Demonstrations), The Association for Computer [15] N. Chambers, D. Jurafsky, Unsupervised learning Linguistics, 2014, pp. 55–60. of narrative event chains, in: Proceedings of ACL- [28] G. K. Savova, J. J. Masanz, P. V. Ogren, J. Zheng, 2007, Association for Computational Linguistics, S. Sohn, K. C. Kipper-Schuler, C. G. Chute, Mayo Columbus, Ohio, 2008, pp. 789–797. clinical text analysis and knowledge extraction sys[16] N. Chambers, D. Jurafsky, Unsupervised learning tem (ctakes), JAMIA 17 (2010) 507–513. of narrative schemas and their participants, in: Pro- [29] Neo4j, Neo4j -graph data platform, 2023. http:// ceedings ACL-2009, Association for Computational neo4j.com/, last accessed 7 March 2023.
Linguistics, USA, 2009, p. 602–610. [30] Z. Shen, K. Lo, L. J. Yu, N. Dahlberg, M. Schlanger, [17] K. Pichotta, R. Mooney, Statistical script learning D. Downey, Multi-lexsum: Real-world summaries with multi-argument events, in: Proceedings of the of civil rights lawsuits at multiple granularities, 14th Conference of the European Chapter of the ArXiv abs/2206.10883 (2022).
Association for Computational Linguistics, 2014, pp. [31] SPOT, The legal innovation & technology lab’s 220–229. spot api, https://spot.sufolklitlab.org, last accessed [18] E. Rahimtoroghi, E. Hernandez, M. A. Walker, March 9, 2023, 2023.
Learning fine-grained knowledge about contingent [32] J. Frank, A2j author, legal aid organizations, and relations between everyday events, arXiv preprint courts: Bridging the civil justice gap using docuarXiv:1708.09450 (2017). ment assembly, W. New Eng. L. Rev. 39 (2017) 251. [19] J. W. Orr, P. Tadepalli, J. R. Doppa, X. Z. Fern, T. G. [33] M. J. Sergot, F. Sadri, R. A. Kowalski, F. Kriwaczek, Dietterich, Learning scripts as hidden markov mod- P. Hammond, H. T. Cory, The British Nationalels, CoRR abs/1809.03680 (2018). ity Act as a logic program, Commun. ACM 29 [20] R. Rudinger, P. Rastogi, F. Ferraro, B. Van Durme, (1986) 370–386. URL: http://doi.acm.org/10.1145/ Script induction as language modeling, in: Proceed- 5689.5920. doi:10.1145/5689.5920. ings of the 2015 Conference on Empirical Methods [34] M. Lauritsen, Q. Steenhuis, Substantive legal softin Natural Language Processing, 2015, pp. 1681– ware quality: A gathering storm?, in: Proceedings 1686. of the Seventeenth International Conference on Ar[21] A. Belyy, B. Van Durme, Script induction as asso- tificial Intelligence and Law, ICAIL ’19, ACM, New ciation rule mining, in: Proceedings of the First York, NY, USA, 2019, pp. 52–62.
Joint Workshop on Narrative Understanding, Storylines, and Events, Association for Computational
Linguistics, Online, 2020, pp. 55–62. [22] L. K. Branting, S. McLeod, K. Megerdoomian,
B. Park, Induction of narrative models for legal case elicitation, in: Proceedings of the Sixth Workshop on Automated Semantic Analysis in Law (ASAIL), 2023. [23] B. Peng, M. Galley, P. He, C. Brockett, L. Liden,
E. Nouri, Z. Yu, B. Dolan, J. Gao, Godel: Largescale pre-training for goal-directed dialog, 2022.
arXiv:2206.11309. [24] J. Lester, K. Branting, B. Mott, Conversational agents, in: M. Singh (Ed.), Practical Handbook of
Internet Computing, CRC Press, 2004. [25] M. Mistica, J. H. Lau, B. Merrifield, K. Fazio, T. Baldwin, Semi-automatic triage of requests for free legal assistance, in: Proceedings of the Natural Legal Language Processing Workshop 2021, 2021, pp.
217–227. [26] K. Megerdoomian, K. Branting, C. Horowitz,
A. Marsh, S. Petersen, E. Scott, Automated narrative extraction from administrative records., in:
AIAS@ ICAIL, 2019, pp. 38–48. [27] C. D. Manning, M. Surdeanu, J. Bauer, J. R. Finkel,