Automated summarization of court trial transcripts can enable lawyers to review and understand cases much more eficiently, but it is challenging for pre-trained large language models (LLMs) in zero-shot settings due to the uniqueness and noisiness of legal dialogue. This is further complicated by the high-stakes of errors, which can mislead readers in a domain where factuality and impartiality are paramount. In this short technical paper, we apply summarization methods to this new domain and experiment with manipulating the transcript text to reduce model errors and generate higher-quality summaries. With human evaluations of metrics like factuality and completeness, we find that zero-shot summarization of trial transcripts is possible with preprocessing, but it remains a challenging task. We observe several open problems in summarizing court dialogue and discuss future directions for addressing them.
1. Introduction ning thousands of pages, making them time-consuming and mentally taxing to read in-full. Lawyers whose work lenges of understanding, retaining, and finding details nested in court dialogue that may have occurred in their distant past or that comes from other attorneys. As collaborators on the present endeavor, lawyers at the Innocence
Summarization of many types of text has been made possible by recent advancements in natural language proels (LLMs): neural models pretrained on vast amounts of text [2, 3]. Previous studies have endeavored to summarize legal text using both LLMs and others in several settings, including abstractive summaries to make legal jargon approachable to laypeople [4], summarizing case outcomes [5], and performing information extraction ing queue of clients waiting to have their cases reviewed been wrongfully incarcerated. The IP has a rapidly grow- from legal texts [6]. However, summarization has not yet been applied to the domain of individual examinafor evidence of a mistrial and other mitigating factors, tions in trial transcripts, and doing so presents technical but the IP’s limited staf are unable to keep up due to the time and efort each lengthy transcript requires.
In this work, we explore how language technologies can be used to automatically summarize examinations in trial transcripts in order to provide lawyers with a concise overview of important points. Summaries that are factually accurate and preserve relevant details could enable lawyers to review transcripts more eficiently and holistically, significantly accelerating their trial review process and enabling the IP to serve more clients. The IP’s social justice work is one example of a high-impact humanitarWorkshop on Artificial Intelligence for Access to Justice (AI4AJ 2023), to pretrained LLMs.
Not only is legal discourse linguistically diferent from Challenges of NLP in High-Stakes Real-World Domains. text scraped from the Web, but trial transcripts also carry LLMs pretrained on vast amounts of Web data have been all the nuances and noisiness of spoken dialogue, and they used to analyze and generate text in a variety of highare furthermore formatted in ways that may seem unnat- stakes domains [2]. However, it remains a challenge to ural to LLMs. Such out-of-domain inputs can exacerbate apply language technologies to real-world settings that language generation problems like factuality errors and are often very noisy and may difer from the data the modsocial biases. In such a high-stakes domain, tools with els were trained on. In the absence of readily available errors can be more harmful than helpful, such as by caus- training data for new domains, prior works have experiing readers to miss important details or influencing their mented with modifying text inputs to optimize zero-shot interpretation of the actual text. Because of the gravity model performance without additional training [8]. For of these potential errors, we rely not only on automatic example, prompt tuning has emerged as a popular way metrics like perplexity, but also on manual human evalu- to improve model outputs for a wide variety of tasks [9]. ation to judge whether generated summaries are truthful However, these works focus on manipulating relatively and relevant. short prompts, whereas we experiment with high-level
This short paper shares some empirical findings in pur- text patterns to make longform court dialogue more unsuit of addressing the above, and specifically contributes derstandable to models. Aside from the dificulties of the following: handling out-of-domain text, text generated by LLMs is prone to problems like social biases, where models per• Assesses the out-of-box performance of a popular petuate stereotypes about gender, race, or other aspects LLM dialogue summarizer on a selection of real of identity [10], and factuality errors, where models halcourt transcript examinations. lucinate false information [11]. Our results demonstrate • Provides human-labeled evaluations of summa- these common pitfalls, and we explore how preprocessrizer outputs on measures of factuality, complete- ing can be used to minimize them and discuss avenues ness, and overall quality. for future work. • Reports on the efects of several dialogue prepro- Summarization in NLP. The goal of summarization is to cessing techniques on these metrics. distill the most important information from long passages • Shares qualitative insights on the summaries that of text. With the rise of neural language models, summamay pave the way for future explorations. rization models have shifted from extractive (identifying important sentences in the original text) to abstractive
Although zero-shot summarization of longform docu- (generating the summary from scratch) and have made ments remains an open challenge, we show that factual, extraordinary performance improvements in summarizcomplete, and helpful summarization of court exami- ing documents ranging from news articles [12] to novels nations is possible with appropriate preprocessing tech- [13]. Most prior work in summarization has focused on niques that manipulate rigidly formatted trial transcripts model design and training, but our work is a zero-shot to sound more like natural language. setting and particularly focuses on dialogue. Dialogue adds new challenges to summarization because, unlike 2. Background and Related Work text written by a single author, it involves multiple participants, frequent coreferences, and a less structured disTrial Transcripts. Trial transcripts in United States courts cussion flow, with some related recent work summarizing follow a consistent high-level structure, though the text written dialogues like chats and email threads [14, 15]. formatting often varies across cases. In general, tran- However, many datasets and benchmarks for summarizascripts primarily consist of dialogue, typically written in tion are constructed in artificial settings: for example, all capital letters as a speaker’s name followed by their the SAMSum Corpus contains abstractive summaries of spoken line, interspersed with descriptive text. Much chats between linguists who were aiming to emulate of this dialogue is comprised of examinations, where a conversations in a messenger app [16]. Spoken converwitness is called to the stand and interrogated by a pros- sations in the real world are studied much more sparsely ecution or defense lawyer. Examinations’ formatting and are even noisier, but a small number of recent works switches to a Q/A pattern: rather than referring to the ex- have begun to explore it [17]. Our work builds on this by aminer and witness by name, they are instead introduced attempting to apply summarization methods to spoken at the beginning of the examination and subsequently dialogue in US courts. referred to as Q and A respectively. These examinations can be of any length—from a few sentences to several dozen pages—and are the portions of dialogue that we aim to summarize.
3.1. Data
5 trial transcripts from which 59 examinations were extracted. The transcripts were provided as scanned PDFs from court proceedings. For each transcript, we use the Google Tesseract library to perform Optical Character Recognition (OCR) and recreate the lines of the transcript as plain-text. The beginnings and ends of examinations are clearly marked on trial transcripts due to a standardized format of court transcripts. Examinations ranged in length from 42 to 6511 words ( = 1563, = 1369) . 3.1.1. Sanitization.
Because of small imperfections in the OCR plain-text conversion, we first sanitized the data by fixing any mistakes manually, including the addition of multiple spaces or newlines where inconsistent. We also removed most procedural text that was secondary to the examination dialogue, typically found following an examiner’s statement of “nothing further” or “no further questions” and which dealt only in court logistics like taking recesses. 3.1.2. Preprocessing. with roles (“The Examiner” or “The Witness”), but this time, for all speakers, we added the word “says” between the speaker and their dialogue, resulting in a format of “<Role or Name> says <Their dialogue>”. The preprocessed lines were concatenated together without newlines into a long paragraph. (Again, we omitted any initial parenthesized text stating the examiner’s name.) • Quote. This condition was identical to the “No quote” preprocessing above, except that we enclosed all spoken dialogue in quotation marks, resulting in a format of “<Name> says “<Their dialogue>””. We wanted to see whether the summarizer would understand speech better when it was enclosed in quotations, as is commonly seen in books and articles, which comprise much of
LLMs’ training data.
Many of the trial transcripts that were furnished were entirely uppercased. Because LMs account for casing when tokenizing text, they treat uppercased tokens as separate tokens from the lowercased versions. LMs tend to see much more lowercased text in their training data, so summarizers tend to do better on lowercased than uppercased text. For all interventions except the control, we lowercased all examinations that were not already truecased before applying any preprocessing techniques.
Preprocessing techniques were applied as interventions on the sanitized data and serve as the chief independent 3.2. Procedure variables in this study. We hypothesized that transforming the unique structure of trial transcript dialogue into We used a large version of BART fine-tuned on CNN data 1 a format more akin to the language that LLMs tend to be as the primary summarizer model for evaluation [3]. We trained on could lead to improvements in summarization chose to use a model in the BART family because of their clarity. In particular, our compared conditions included: popularity and ubiquity on natural language generation tasks, and this particular fine-tuned model is one of the • Control. Nothing about the examination was most widely used for the task of summarization. As this changed before it was summarized; any Q/A tags model is already fine-tuned for summarization, we did remained as is, and each speaker’s dialogue ended not engineer any prompt to accompany the text passed with a newline. in from an examination. Other models exist that have • Speaker. In an efort to give the summarizer previously performed well on summarization, which we more information about the speaker, we replaced briefly compare: T5 2 [18] and BART3 [3], both finetuned the Q/A tags with the participant’s role in the on the SAMSum corpus. However, there did not seem to examination—“The Examiner” or “The Witness” be drastic diferences between the summaries and perrespectively— resulting in a format of “<Role>: plexities of the BART-CNN model and others, so we chose <Their dialogue>”. (Occasionally, other speak- to focus primarily only BART-CNN for this paper and the ers may interject during the back-and-forth be- efects of difering preprocessing techniques. We leave tween the examiner/Q and the witness/A; we left experiments with additional models for future work. those speakers as is. We also omitted any initial Setting summary lengths. For examinations shorter parenthesized text stating the examiner’s name, than twice the model’s maximum output summary length which sometimes appeared before their first spo- of 142 tokens, the maximum summary length was set ken line.) to half the length of the examination and the minimum • No quote. Since the LLM we used was finetuned summary length was set to a quarter of the length of on news articles (see section 3.2), we attempted to preprocess the examinations to mimic quotes in 21““fpahcielsbcohomk/ibda/flarnt--lta5r-gbea-scen-nsa”monsuHmu”gogninHgFuagcgeingFace news articles. We once again replaced Q/A tags 3“philschmid/bart-large-cnn-samsum” on HuggingFace the examination to prevent the generation of summaries that were of a similar length or longer than the examinations themselves. For examinations longer than the summarizer’s 1024 token input maximum, the examination was split into “chunks” just below the summarizer’s maximum input length without splitting a sentence. The very last “chunk” of text was prefixed with text from the previous chunk to provide context for short inputs and prevent summaries that were longer than their inputs. Each chunk was then summarized individually and concatenated together. 4
For the particularly long examinations, this “chunking” method resulted in very long summaries, so any summaries over 400 tokens in length were repeatedly re-summarized until they were under 400 tokens. This was not common, and when it was necessary it almost always only took one re-summarization. Pursuant to our goals with these summaries, we hoped this would produce summaries that were brief enough to provide a quick overview of the examination’s content that a lawyer could read quickly.
Generating and evaluating summaries. For each extracted examination under each preprocessing condition, the summarizer was applied with the above constraints on summary length. We compiled all generated summaries, and each examination along with its 4 summaries was assigned to two human judges. The human judges were asked to rate summaries based on the metrics described in the following section. 3.3. Analyses
3.3.1. Metrics.
scriptive metrics were recorded for each summary:
For each summary generated from the examinations, two human judges provided their subjective assessment on the three metrics above. They were asked to first read • Perplexity, assessed first comparing the sum- the unsummarized examination in full and then read/rate maries of the BART-CNN model with the perplex- each summary created from it so that the examination’s ity computation from GPT-2 [19] using a sliding details would be fresh in-mind. window technique with a stride of 512 tokens, and again using the perplexity computed from 3.3.2. Statistical Tests. each summarizer variant (i.e., BART-CNN, BARTCNN-SAMSum [abbreviated to BART-SAMSum], and T5) [20]. Perplexity is typically used to evaluate language models, but it can also be used to get an idea for the quality of generated text by
Because the same examination was used as input to each of the summary conditions, we performed a 4-way repeated measures ANOVA for each of the dependent variables (Perplexity, Lexical Overlap, Factuality, Completeness, and Overall Quality) to detect diferences between groups and performed Bonferroni correction for multiple 4fTrhoemtoHkuegngizienrguFsaecde’fso“rfacocembpouotkin/bgaerxt-abmasine”attioonmlaetncghththsewtaoskleonaidzeedr comparisons ( = .008). For the metrics from human used by the summarizer model. To determine the length of sum- judges (Factuality, Completeness, and Overall Quality), maries, we used SpaCy’s tokenizer. we first converted Boolean answers of True/False and quantifying how “confused” a typical LLM would be about the text. • Lexical Overlap, assessed by finding the lexical overlap between the summary and the top 20% most frequently occurring tokens (excluding stopwords) in each examination. We report this as a ratio of words that were retained in the summary over the number of frequently occurring tokens. In principle, this metric could assess the balance the summarizer struck between being abstractive vs. extractive, as well as how true the summarizer stays to the examination’s language and most common discussion points.
Central to validation of summarizers in the domain of court transcript review, we also examined several aspects of summary quality that required human examination: • Factuality, a Boolean assessment of whether or not all of the summary’s stated accounts of the examination are faithful to the original text. If even a single statement, attribution, name, or pronoun ran counter to fact, that summary was not considered factual. • Completeness, a Boolean assessment of whether or not the summary mentioned all of the important events in the examination. If even a single essential detail of the examination was omitted, that summary was considered incomplete. • Overall quality, a Boolean assessment of whether or not the summary was interpretable enough to obtain a gist for the examination. It was possible for a summary to be factual and complete, but e.g., discuss additional non-sequiturs or arrange the sentence structure poorly so that meaning was obscured, and would thus be perceived as poor quality.
average rating for each summary. To examine the degree to which subjective interpretation of the summaries affected perceptions of quality, we also computed Cohen’s
which describes the proportion of agreement between raters above and beyond chance [21].
ror the efects described previously. However, again due both the control and speaker conditions. The quote
One- to the number of comparisons after Bonferroni corrections, none of these efects would be significant in this
.001, 2 = .45. After Bonferroni correction, all but two comparisons were significantly diferent from one another, < .002 , (see Fig. 3). Specifically, factuality ratings in the speaker condition were only marginally lower than the no quote condition, = .009 . Additionally, there were no significant diferences in factuality ratings between the no quote and the quote conditions, = .874 .
Completeness ratings.
There were significant differences in completeness ratings across conditions, 2 = .060. After Bonferroni (3, 174) = 3.69, = .13, correction, only two comparisons demonstrated significant diferences. Specifically, the speaker condition had significantly higher completeness ratings than the quote ( = .003 ) and the no quote ( = .008 ) conditions.
Overall Quality ratings. There were significant differences in overall quality ratings across conditions, (3, 174) = 7.88, < .001, roni correction, only two comparisons demonstrated significant diferences. Specifically, the control condition had significantly lower overall quality ratings than the speaker ( < .001 ) and the no quote ( = .001 ) conditions. 2 = .120. After Bonfer
objective report, we discovered several themes in summary quality that bear mentioning, and may be of use for future studies.
Exemplar Summary. Many summaries provided excellent synopses of the dialogue’s contents, including the following that condensed an examination that was 590 words: “The witness is a senior criminalist with the orange county sherif’s crime lab. The witness is asked to examine a knife found at the scene of a murder. The knife is a buck-style knife with a brown plastic piece on either side of it. The Witness says he did not find any trace elements of blood or bodily fluids.”
picts the contents, it does misrepresent the gender of the witness, leading to a pervasive mistake:
Gender Bias. Through qualitatively studying generated summaries, we observed an explicit male-gender bias: many summaries defaulted to assuming actors were men rather than women, even when the original examination text was explicit in referring to an actor with feminine titles like “ma’am.” This asymmetrical representation of men and women is not a novel phenomenon; gender bias has been well-documented in many LLMs [10].
Repetition. Sharing a snippet from a summary that was marked as factually accurate and complete, the output still lacks some readability due to repetition of actor nouns: “The Witness says he has known the boy since he was in his mother’s womb. He says he knows the boy because he knows his family. The Witness says the boy is not in a gang. The witness says he’s never heard of the boy being a gang member.
Hallucinations. Hallucinations that obviously misrepresent the examination content are arguably of less concern for users because they are more likely to be caught by readers compared to subtle perturbations of court facts. The following examples demonstrate the absurdity of such dramatic hallucinations: “A man was shot in the head by a colleague in a New York City ofice. The shot was fired by a member of the jury in the trial. The gunman was standing in the same position as the shooter. A man was taken to jail for a photo shoot. He conditions significantly increased the perplexity comsaw a photo of a man he thought looked pared to the control, these approaches led to significant like him.” improvements in factuality, completeness, and overall quality, showing that perplexity is not necessarily a re
Some hallucinations also demonstrate sensitivities to flection of summary quality. Additionally, interrater relithe fine-tuning training set and the efects of hyper- ability was fairly uniform in condemning the quality of compression from re-summarizing long examinations, the control condition’s summaries, but was surprisingly with the following example mentioning a commonly- lower across preprocessing conditions. This highlights referenced figure in the contemporary news who was yet another dificulty of assessment for summaries in the plainly not a party to the case being summarized: court dialogue domain: subjective disagreements over what constitutes good summaries and/or omission of key “A fight broke out between Edward Snow- details. den and a group of friends after he tried Limitations and Future Directions. This study’s human to leave the house.” raters were not lawyers, who may have had feedback on the subjective measures and better expertise on how
Other hallucinations are almost understandable con- helpful a summary would be in practice. Future work sequences of the quirks of spoken language, herein pro- should iterate with lawyers to develop more fine-grained ducing a summary mentioning two characters with nick- criteria for what makes a summary “good” or “bad,” and names “Rock” and “Blue Dog:”5 by providing continuous, rather than binary, ratings of “The court asks the witness if he or she success; e.g., determining how many important facts has ever made an arrest of a dog. The were omitted rather than whether or not any were. witness tells the court he has never seen Additionally, we only performed subjective rater coma dog in his life. The court asks if the parisons on summaries from one model, BART-CNN. witness has ever seen a rock in his or her Though we briefly tried out other models, including T5 life. He says he has, but he doesn’t know and a BART model fine-tuned on the SAMSum corpus, if it was a dog or a rock.” we found only minor perplexity diferences and little tangibly diferent in summary outputs; however, exper
As Figure 3 demonstrates, there is much room to im- imenting with models with diferent architectures and prove these summaries, and repairing the qualitative is- training datasets could improve zero-shot summarization sues above may likewise improve factuality, complete- performance, especially with more modern generative ness, and perceived overall quality. models. In the big picture, this work demonstrates that while LLMs are powerful, they may not be able to keep track of facts reliably. This motivates work on NLP ap5. Discussion proaches that can store information in a more consistent and interpretable way than black-box LLMs, such as with maintaining state graphs and more recent chain-ofthought techniques [22].
Lastly, this work may expand avenues for novel application of court dialogue summary, including: as a learning tool for law-students to either evaluate or produce summaries, as an avenue for increasing public literacy of court proceedings by providing summaries stripped of legal procedure, and as a possible novel benchmark for domain-specific LLM adaptations in preserving the factuality and completeness of summarized text.
Efect of Preprocessing. Factuality errors were present to some degree in all four preprocessing conditions, but all forms of preprocessing helped to improve factuality, completeness, and overall summary quality over the control. This is likely because preprocessed examinations more closely resembled the text that BART was trained on, and suggests that manipulating the input text may be a way to boost summarization quality. However, there seems to be a tradeof between factuality and completeness: the Quote and No Quote conditions’ propensity to produce more extractive than abstractive summaries led to improved ratings of factuality, but sufered in terms of completeness compared to the Speaker condition.
Challenges with Evaluation Metrics. Measuring summarization quality is challenging because neither quantitative nor qualitative metrics are perfect, and they sometimes contradict each other. Although the preprocessing
Our empirical results suggest that automated summarization of raw legal examinations yields poor quality summaries, but that this can be improved by preprocessing the court dialogue to better resemble the natural language that LLMs were pretrained on. These approaches still leave large gaps in the factuality and completeness of 5Note: this summary comes from an output lacking any preprocessing; in each of the preprocessing conditions, the nickname ambiguity was avoided. summaries, and their perceived quality is volatile. Nevertheless, this work may serve as a motivating recipe for manipulating court examinations to achieve reasonable summarizations in a zero-shot setting, an approach that may be practical due to the domain’s sparsity of finetuning data and could potentially make lengthy transcripts easier for lawyers to review.
Petersen (a lawyer working with the Loyola Law School’s Project for the Innocent) for his feedback and guidance on this project, as well as the eforts of several research assistants working on the Briefcase6 team who aided with the subjective summary quality metrics: Saad Salman, Tanya Nobal, Jennifer Siao, and Evan Sciancelapore.
view/17723. doi:1 0 . 1 6 0 9 / a a a i . v 3 5 i 1 6 . 1 7 7 2 3 . [18] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-totext transformer, The Journal of Machine Learning Research 21 (2020) 5485–5551. [19] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9. [20] H. Face, Perplexity of fixed-length models, 2023.
URL: https://huggingface.co/docs/transformers/ perplexity. [21] J. R. Landis, G. G. Koch, An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers, Biometrics (1977) 363–374. [22] B. Wang, X. Deng, H. Sun, Iteratively prompt pretrained language models for chain of thought, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 2714–2730.