This paper critically examines some recent studies of the legal-reasoning capabilities of generative AI. It also discusses which roles traditional symbolic approaches can have in the era of generative AI. The introduction of ChatGPT by OpenAI in November 2022 was a 'big bang' in AI. Never before was an AI tool available for so many and so easy to use for so many diferent tasks. The ease with which it generates flawless natural language for a wide variety of tasks such as summarising documents, writing essays about any given topic, writing poems, drafting travel plans, outlining presentations, and even solving computer-programming exercises is amazing. And all this essentially with the simple technique of predicting the most likely next word in a sequence of words. It is therefore easy to think that traditional symbolic AI research on reasoning and argumentation is now obsolete and that the right way to let the computer engage in reasoning and argumentation is by using generative AI founded on large language models.
CEUR ceur-ws.org
2.
Then we make some methodological observations in Section 3 and review recent experiments in applying LLMs to legal reasoning in Section 4. We then discuss what the field of computational argumentation can learn from these studies in Section 5, after which we conclude. https://webspace.science.uu.nl/~prakk101/ (H. Prakken)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Argumentation is “…the giving of reasons to support or criticize a claim that is questionable,
or open to doubt” [1, p. 285]. The field of AI & Law has developed formal and computational
models of legal argumentation since [
Case-based approaches were initially developed to account for the fact that in Anglo-American
jurisdictions case law instead of legislation traditionally is the main source of law, where courts
have to decide new cases by drawing analogies to decided cases. Case-based approaches have to
account for the fact that cases often not just have similarities but also diferences. Seminal work
was on HYPO [
While early on rule- and case-based approaches were presented as alternatives, later the
awareness arose that they complement each other, since case-based reasoning often is about
whether a rule’s conditions are satisfied. A challenge for rule-based accounts is that the
conditions of legal rules are often vague and general, and no clear rules can be given for when
they are satisfied. Here case-based approaches can complement rule-based approaches by
providing forms of argumentation for interpreting legal concepts [
In sum, AI & law has developed rich models of various forms of legal argument, including rule-based, case-based and value-based accounts, which draw on various sources, including legislation, case law and social and moral value considerations. Moreover, all this work takes a knowledge-based approach: the required knowledge is encoded in a symbolic form that is understandable for the machine and the computer reasons with it in a formally defined way, ideally based on the laws of logic and rational reasoning. The advantages of this approach are transparency and explainability: humans can see which knowledge the machine uses and the machine can explain its outcomes by showing how it reasoned with this knowledge. A big disadvantage of this approach is that it is often hard to acquire and represent a suficient amount of knowledge in a form that can be manipulated by the machine. This is the notorious knowledge acquisition bottleneck. Hence the attractiveness of large language models as a means to generate legal argument.
Evaluation of knowledge-based AI applications can be done at three levels: evaluating the inputted knowledge, evaluating the reasoning mechanism (for instance, on whether it implements some philosophically acceptable model of rational reasoning, or on soundness and completeness properties with respect to such a model) and evaluating the output. When evaluating applications of generative AI, evaluation at the first two levels becomes hard so often only the system’s output is evaluated. Moreover, this output is natural language instead of some formal language, with all the ambiguity and vagueness that comes with it, so interpreting the output is not always an easy task. In consequence, evaluation studies of generative AI are inherently experimental, often statistical and can involve subjective elements.
A well-known drawback of LLMs is that there is no connection between the large statistical language model learned by the LLM and reality. All that an LLM ‘knows’ is how often words go together in similar contexts. This often causes a LLM to ‘hallucinate’ facts. While this may not be a problem for creative applications like writing prose or poetry, it is a serious problem when an LLM is asked to produce high-quality information or arguments in high-stake contexts; and legal contexts are often high-stake.
There is much research on addressing this problem. Many of them involve prompt engineering, that is, applying ingenious ways to write the prompts that are the user input of LLM applications. Zero-shot prompts do not contain any examples of desired output but directly ask a question or specify a task. Few-shot prompts do provide such examples. Chain-of-thought (CoT) prompting consists of ways to ask the model to ‘think’ step-by-step rather than solving a complex problem at once. Zero-Shot CoT does just that while few-shot CoT methods combine it with examples of desired output. Such prompts are often formulated as problems of pattern completion, which consist of showing the model a pattern of expected answers: this increases the probability that the model will indeed give an answer in terms of this pattern. Yet another way in which prompts can be engineered is to include one or more documents that have to be taken into account in the model’s answers. When these documents are retrieved from other sources after entering a prompt, this is called retrieval-augmented generation. In legal applications it makes sense to include or retrieve legislation or case law.
In this paper we will review all serious recent studies of the legal-reasoning capabilities of generative AI that we know of, with among other things attention for their prompt-engineering methods. We will ask the following questions about the reviewed studies.
• Which reasoning capability is tested and according to which reasoning model? • How direct was the testing? Were proxies for reasoning abilities used? • Which method of prompt engineering is used? • How systematic is the evaluation? Is it subjective or objective, qualitative or quantitative? • What is compared? LLMs or prompting methods against each other or also against human performance?
All reviewed studies study tasks that involve legal reasoning, though in diferent ways. Some studies are about making exam questions, other studies involve specific reasoning tasks (mostly rule application) and some studies are about the generation of legal documents that typically contain argumentation. Many studies apply or refer to the IRAC method of legal reasoning, popular in Anglo-American legal education. IRAC stands for Issue-Rule-Application-Conclusion. Here, Issue is the task of determining the legal issue of a case, Rule is the task of identifying the relevant legal rules (which can also be precedents), Application is the task of determining how the rules should be applied to the facts, and Conclusion is the task of drawing a legal conclusion from the rule application. While in reality issue spotting can be far from trivial, in all studies reported below the issue is in fact given. Then we see that the IRAC model in fact abstracts from all AI & law models of legal reasoning discussed above in Section 2.
There are a few studies on legal document generation.
Perlman [
Iu & Wong [
In sum, both these studies have no explicit testing of reasoning capability, so testing is indirect, the prompts were zero-shot, evaluation is unsystematic and subjective, and no comparisons are made. Because of the lack of explicit and objective evaluation standards, these studies cannot provide valid and reliable results, though they can have heuristic value.
Trozze et al. [
In sum, the prompt did not give any reasoning instructions, and no explicit testing of reasoning capability. Testing is thus indirect, with as proxy how often ChatGPT agrees with the human lawyer. Systematic quantitative evaluation but no real comparison with human performance, since the human lawyer’s performance is used as the evaluation standard.
Several studies let the models take legal exams or answer exam questions. These studies mostly indirectly evaluate legal-reasoning capacities, since many exam questions do not directly test the student’s reasoning or argumentation skills. These studies can only be regarded as evaluating such skills on the assumption that successfully making legal exams requires such skills. This may not always be the case since questions can also test for the possession of legal knowledge.
Yu et al. [
In sum, explicit testing of reasoning abilities so testing is direct, zero-shot and few-shot prompting, some prompts ask to apply a particular mentioned but undefined reasoning method, systematic quantitative evaluation, comparisons between prompting methods and with other NLP methods but not with human performance.
Choi et al. [
In sum, no explicit testing of reasoning capability/model, except for a basic form of CoT prompting for the multiple-choice questions. Testing is indirect, with the exam scores as proxy for legal reasoning abilities. Systematic quantitative evaluation in terms of exam scores, comparisons between prompting methods and with human performance.
Katz et al. [
In sum, no explicit testing of reasoning capability. Testing is indirect, with exam score as a proxy for legal reasoning abilities. The zero-shot prompts correspond to the exam questions. Systematic quantitative evaluation and comparisons with human performance.
Nay [
In sum, implicit testing of reasoning capability, (deductive rule application without chaining). Testing is indirect. The prompts provide relevant legal information but give no information about or explicit examples of the expected reasoning. Systematic quantitative evaluation and comparisons between prompting methods but no comparisons with human performance.
Jiang & Yang [
In sum, explicit testing of reasoning capability, namely, the legal syllogism (deductive rule application without chaining). Testing is direct. The prompts contain an explanation of the reasoning method and ask to apply it. Systematic quantitative evaluation and comparisons between prompting methods but no comparisons with human performance.
Similar work is Deng et al. 2023 [
In sum, explicit testing of IRAC-like reasoning capability. Testing is indirect in terms of predictive performance. The breakdown into four prompts corresponds to IRAC-style reasoning. Systematic quantitative evaluation and comparisons between prompting methods but no comparisons with human performance.
A limitation of both [
Kang et al. [
In sum, explicit testing of reasoning capability, namely, IRAC. Testing is direct, since ChatGPT is evaluated on how well it can reproduce pre-encoded IRAC structures. Various zero- and few-shot prompting methods are used, giving less or more of the desired solution. Systematic quantitative evaluation, comparisons between prompting methods but not with human performance. An important thing to note is that quite some structure is added to the prompts.
Blair-Stanek et al. [
In sum, explicit testing of reasoning capability, namely, deductive rule application with chaining, with awareness that the model might apply implicit knowledge of the statutes. Testing is direct. The prompts ask to do ‘Entailment/Contradiction reasoning’ but do not explain what it is. Systematic quantitative evaluation and comparisons between prompting methods but no comparisons with human performance.
Guha et al. [35] present the LegalBench legal reasoning benchmark for six legal tasks corresponding to the stages of the IRAC model plus two related tasks. The datasets for the six tasks are restricted to clear cases with objectively correct answers. The authors then apply various LLMs to these tasks, where the prompts contain between zero and eight example answers and an instruction to the LLM to explain its reasoning. For all tasks, GPT-4 performed the best, with accuracies between 59.2 and 89.9%. The authors note that their experiments should be seen as providing lower bounds on performance since they see considerable scope for improvements.
In sum, explicit testing of reasoning capability, namely, IRAC with chaining. Testing is thus direct. The prompts can contain examples of the expected reasoning and give the instruction to explain the reasoning. Systematic quantitative evaluation and comparisons between LLMs in terms of accuracy but no comparisons between prompting methods or with human performance.
Trozze et al. [
In sum, the prompt asks to apply the IRAC method but no explicit testing of whether it was applied. Testing is indirect, with as proxy how often ChatGPT mentions a law also mentioned in the case. Systematic quantitative evaluation but no comparison with human performance.
Servantez et al. [36] propose an IRAC-inspired prompting method called ‘Chain of logic’. Each prompt contains an example of a rule, a fact pattern and an issue, the rule’s decomposition in elements (the conditions and the conclusion) and a formalisation of the rule in proposition logic. Then the example answers each rule element separately, gives the logical expression for the conditions yielded by the answer, and resolves it to give the final answer. Thus the model should in one shot learn to apply this IRAC-style process from the example. The authors apply this to several rule-based tasks from the LegalBench legal reasoning benchmark [35]. They apply five large language-models including GPT-4 and compare the accuracy of their prompting method to several zero- or few-shot prompting methods. Their method outperforms all other methods for all LLMs, although not by wide margins. With GPT-4 they obtain 92.3% accuracy, while the worst-performing method scores 86.3%. The authors conclude that, compared to the literature, their method is the only few-shot method that consistently outperforms zero-shot prompting. Limitations of this study are that the rules in LegalBench are simpler than in reality and that the method only seems to work for single-step rule-application.
In sum, explicit testing of reasoning capability, namely, IRAC-style deductive rule application without chaining. Testing is direct. The prompts give a detailed example of the expected reasoning. Systematic quantitative evaluation and comparisons between prompting methods in terms of accuracy but no comparisons with human performance.
In this section we discuss what can be learned from the preceding overview. Here it should be taken into account that studies that are only published on ArXiv are presumably not peer reviewed.
The studies involving exams and document-generation tasks do not explicitly test some reasoning capability, which makes it hard to draw firm conclusions from them on such capabilities, since they do not distinguish between the possession of legal knowledge and the ability to apply it. The other studies do explicitly test on reasoning capabilities and test some form of deductive reasoning with legal rules, often structured in terms of the IRAC model. Some studies do not explain the reasoning method they ask for while other studies explain them with examples. A commonplace in both legal philosophy and AI & law is that deductive rule application is far too simplistic as a full model of legal reasoning. The exam and document-generation studies could implicitly test full-fledged argumentation capabilities, including the use of case- or value-based reasoning and the consideration of conflicting arguments. However, whether they do is hard to tell from the publications. This is a point on which computational models of legal argument could be useful, namely, as standards for the argumentative outputs of legal generative AI.
Most studies that make comparisons do so between several prompting methods or several
LLMs. Two studies compare between AI and human performance, namely, [
Some studies include reasoning instructions of varying levels of detail in the prompt and/or
verify to what extent the model’s output obeys these instructions. A general trend in the results
is that such prompting methods improve performance but not consistently. Moreover, there are
some methodological pitfalls here. The first is memorisation. Questions (for instance, bar exam
questions) may be in the training data, so the model may have seen them before, or the model
may in other ways have applied ‘shortcuts’ included in its statistical language model. Some of
the discussed studies show awareness of these issues [
Next, even if a model structures output according to some reasoning method, it may be that the model has not followed the method. Striking examples are reported by [37], who found that GPT-3.5 when used with CoT prompting does not always behave according to the reason it says it applied. A simple example is with multiple-choice questions with two options A and B. When GPT-3.5 is only shown examples with A as the correct answer, it then tends to prefers answer A and gives a reason for A even if B is the correct answer. Thus the reason GPT-3.5 gives for its answer is not the reason it applied. More worrying examples involve racial and gender biases.
It might be argued that in legal applications this is not a serious problem since in the law all that matters is the justification as it is given, since that is by which the parties, appeal courts and the general public can assess the quality and acceptability of a decision. In philosophical terms, it is not the context of discovery but the context of justification that matters. However, against this it can be argued that when alternative decisions are legally acceptable, it is still undesirable that the choice for a particular decision and for which arguments and evidence to include in a decision is influenced by bias. This arguably holds the more for texts that do not contain decisions but standpoints of the parties, such as summons, complaints or briefs.
Regardless of this discussion, another way in which symbolic computational models of legal argument could be useful is in formulating reasoning instructions in the prompt. A natural idea is to formulate few-shot or CoT prompts in terms of some theory of rational reasoning or decision-making. It might be said that (legal) prompt engineering is applied (legal) philosophy.
Research on legal-reasoning capabilities of generative AI is rapidly emerging but still inconclusive as regards quality or practical usefulness. If reasoning models are made explicit, then they are (almost?) always some simple deductive form of rule application, which is generally regarded as too simplistic as a full-fledged model of legal argument. The possible roles of symbolic computational models of legal argument are threefold: as guidance for prompt engineering, as standards for evaluating outputs of legal-generative AI, and as symbolic alternatives to legal-generative AI, possibly combined with the latter as conversational interfaces. In any case, traditional symbolic AI research on legal reasoning and argumentation is not yet obsolete. [35] N. Guha et al., LEGALBENCH: a collaboratively built benchmark for measuring legal reasoning in large language models, 2023. ArXiv:2308.11462. [36] S. Servantez, J. Barrow, K. Hammond, R. Jain, Chain of logic: rule-based reasoning with large language models, 2024. ArXiv:2402.10400. [37] M. Turpin, J. Michael, E. Perez, S. Bowman, Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting, in: Advances in Neural Information Processing Systems, volume 36, 2024.