Lawyers can analyze and understand specific situations of their clients to provide them with relevant legal information and advice. We qualitatively investigate to which extent ChatGPT (a large language model developed by OpenAI) may be able to carry out some of these tasks, to provide legal information to laypeople. This paper proposes a framework for evaluating the provision of legal information as a process, evaluating not only its accuracy in providing legal information, but also its ability to understand and reason about users' needs. We perform an initial investigation of ChatGPT's ability to provide legal information using several simulated cases. We also compare the performance to that of JusticeBot, a legal information tool based on expert systems. While ChatGPT does not always provide accurate and reliable information, it acts as a powerful and intuitive way to interact with laypeople. This research opens the door to combining the two approaches for flexible and accurate legal information tools.
1. Introduction language interface to many types of knowledge. In the legal domain, an important role of lawyers is to provide ChatGPT1 has garnered significant attention from the information and advice about legal problem to laypeople. public, academia, industry, and media. It is able to per- However, for many people, the related expenses of hiring form a huge variety of textual tasks after simply being a lawyer can pose a significant obstacle to safeguarding asked through a free chat interface. The model can fur- and enforcing their rights. They find themselves living ther be accessed using an API (Application Programming in a "legal advice desert"[2], unable to find the means Interface)[1], which gives developers the ability to create to safeguard their rights. Despite the important work products enhanced by ChatGPT across many diferent done by scholar and practitioners in increasing access to areas. Every day, dozens of applications using this API justice[3], many individuals still face issues with it.[4] are launched.2 Artificial Intelligence (AI) has the potential to play a
While ChatGPT’s performance in carrying out nat- significant role in promoting access to justice. AI could ural language conversations is impressive, its potential serve many parties simultaneously and provide legal inapplications are not limited to that. In addition to general- formation to parties in an eficient, accurate, and costpurpose applications such as search engines,3 it could efective manner. A number of products in this area have also be used in domain-specific tasks, such as in the le- been developed and used with success.[5] gal domain. ChatGPT can be seen to provide a natural In this research, we sought to explore the performance of ChatGPT in providing legal information, compared to self-help tools that specialize in the legal domain. To this end, we conducted an experiment on ChatGPT, utilizing simulated legal cases to qualitatively evaluate its performance. We compared the responses to the JusticeBot, a legal decision support tool focused on landlord-tenant disputes.
Our investigation approached the provision of legal information as a multifaceted process, comprising not only the acquisition of precise and reliable sources of information but also the ability to elicit an accurate description of users’ legal predicaments and needs, the aptitude to discern relevant and extraneous facts from the users’ input, and the skill to furnish pertinent legal information relevant to the users’ situation via legal reasoning. Our research thus underscores the vital role played by each of these components in ensuring the efective delivery provide legal information, understand the strengths of significantly improves ChatGPT’s performance.[ 21, 22] ChatGPT, and look at opportunities for integration of Chain-of-thought prompting[23] has been shown to be ChatGPT and other legal tools in the future. efective in improving performance in the legal field as well.[24]
In addition to testing ChatGPT using exam sets, re2. Related Work search has been conducted on comparisons between ChatGPT and human experts, and corpus-based evaluations of ChatGPT performance in the legal field have been conducted.[25]The study found that ChatGPT-generated answers were generally evaluated as more helpful than humans’, but it also found that ChatGPT may fabricate facts, especially in the legal field, where ChatGPT may quote non-existent legal texts to answer questions.[25] The study compared the answers generated by ChatGPT to answers taken from Wikipedia and Baidu Baike4 Due to the specificity of the legal field, these answers may not always be accurate [26].
In this work, we prepare simulated cases, and interact with ChatGPT as if we were layperson parties of these cases. We compare the answers given by ChatGPT to the answers given by JusticeBot, a legal information tool developed at the Cyberjustice Laboratory, created by legal experts.
Using AI and technology to increase access to justice by giving legal information has been an important field of investigation. For example, Branting et al built a tool to inform individuals of their rights regarding protection orders [6]. Zeleznikow’s GetAid system aims to help lawyers determine whether an individual is eligible for legal aid [7]. The Loge-expert system aimed to help people understand their housing law situation using an expert system [8]. Housing law questions were also investigated using machine learning in [9]. In [5], the authors suggest the "JusticeBot" approach to build legal decision support systems for laypeople.
Here, we examine whether ChatGPT could be used to provide legal information to increase access to justice. This would have the advantage of not requiring the manual adaptation to new legal areas. We compare the answers given by ChatGPT to answers given by a JusticeBot tool focused on landlord-tenant disputes [5].
Experiments on ChatGPT’s performance in the le- 3. Access to Justice gal field have been conducted extensively, according to studies testing ChatGPT’s performance on the U.S. Laypeople often have trouble resolving their everyday lebar Exam,[10] While ChatGPT placed in the bottom gal disputes. A majority of people will at some point have 10% of law students, the improvide GPT-4 model was to deal with a legal issue, such as neighbourhood, employestimate to pass the bar and place in the top 10% of ment or debt problems. [27, 28] However, it can be very students.[11, 12] Another study tested ChatGPT’s perfor- tricky to resolve such disputes. Studies conducted have mance on law school exams, showing that it was able to shown that only around 20% of legal problems that arose pass the exam, but could only be considered a "mediocre in the past three years had been resolved [29]. These law student".[13] It was also found that ChatGPT has issues are likely to be especially pronounced for individsome ability to write legal documents.[14] Further, GPT- uals who do not have access to professional legal help, 4 was found to display strong performance when given which can be quite expensive. Such individuals may not legal annotation tasks to carry out [15, 16] and when be aware of which rights apply to them, which forms explaining legal concepts using case law [17]. In the they need to fill out [ 30], or what the relevant facts are context of access to justice, the GPT-4 model further regarding their case [31]. showed promise in intervening in an online dispute reso- Self-represented litigants, lacking precise legal inforlution context, by reformulating inflammatory messages mation guidance, may navigate through the labyrinth or even autonomously suggesting interventions to medi- of litigation like a ship without a compass,[32] and may ators [18]. ifnd themselves at a disadvantage due to missing critical
Remarkably, ChatGPT exhibits a certain level of le- information,[33] hindering access to justice and undergal reasoning ability despite not having been specifi- mining public confidence in the judicial system. The imcally trained on legal data, owing to the emergent abil- pacts may be especially grave when laypeople encounter ities derived from the scaling of the model.[19] This a party that is well financed and has previous experience suggests that ChatGPT is a general-purpose model, it in going through litigation.[34] Such a party imbalance contains laws about human common sense, a capa- may be present, for example, in housing disputes. [32] bility that, when combined with specialized domain A key aspect in addressing this situation is providing knowledge,[20] has the potential to solve the challenges individuals with legal information [35]. As was discussed encountered in legal AI training in the past. We can enhance its performance in the legal domain through prompting. Yu et.al showed that such legal prompting 4Baidu Baike is a Chinese-language collaborative online encyclopedia owned by the Chinese technology company Baidu. It’s considered to be the "Chinese version of Wikipedia". in section 2, the internet and artificial intelligence have range of lay people, and every minor information error been important components of such self-help tools in the may potentially lead to harmful decisions being made by recent past. Such tools are able to take a more interactive, laypeople. and personalized approach than e.g. books providing legal information.
Such tools are generally focused on providing legal 4. Methodology information. This refers to more general principles and rules. Legal advice, which focuses on the unique circum- In order to qualitatively evaluate the capability of Chatstances of a particular [36], can usually exclusively be GPT to provide legal information, we first determine provided by members of the legal bar [37]. criteria that we will use to evaluate the accuracy and
Some legal tasks, such as negotiating with the other trustworthiness of information provided by ChatGPT, side and predicting the outcome, likely constitute the and how well it interacts with users (section 4.1. Then, practice of law [38]. For other tasks, this line is not as we use ChatGPT to generate simulated cases (section 4.2). clear, which may cause dificulty for the development of Finally, we interact with ChatGPT as if we were parties legal self-help tools. in the simulated cases without legal training, and ask
As the accuracy and relevance of legal information pro- ChatGPT to provide legal information (section 4.3). vided by legal self-help tools increases, users can access We try the same cases in the JusticeBot tool,5 a legal legal provisions, relevant cases, from these tools that are decision support tool developed at the Cyberjustice Labclose to the legal issues they face, and users can use the oratory and focused on landlord-tenant disputes. The above information to choose the next step to take. The JusticeBot was built in collaboration with the Tribunal line between the dichotomy of legal information and legal Administratif du Logement du Québec, the housing triadvice is gradually blurring, and the content included un- bunal in Québec, and with financing from the “Ministère der the concept of legal information is becoming broader. de l’Économie et de l’Innovation Quèbec”, and has been The expansion of the scope of legal information can lead accessed by over 20k users since being launched in the to conflicts between these two concepts. summer of 2021.
We need to be cautious when defining the scope of JusticeBot is based on an expert system methodology, legal advice to prevent excessively limiting the potential where all content has been created by a legal expert, of helping people.[33] The objective of restricting entities and the system is fully deterministic to ensure the accuthat provide legal advice is to protect parties from being racy of the provided information [5]. ChatGPT, on the disadvantaged by erroneous legal advice, rather than other hand, is based on large language models, which are turning legal advice into a privilege that obstructs parties trained on enormous corpora of texts to absorb patterns from comprehending the rules of the society in which from it. they reside. ChatGPT, on the other hand, is based on a large lan
Providing individuals with legal information can help guage model, which is trained on a enormous corpora of them better understand their legal situation. This is an text data. By predicting the next possible sequence of text, important step in making them aware that their issue it absorbs patterns from the data. The scale of its training has a legal solution, and how to enforce their rights with makes it "emerge" remarkable abilities in many textual regards to this situation. Of course, building tools that tasks. Comparing these tools will allows us to better uncan provide such information can take a lot of efort, in- derstand the trade-ofs of these diferent approaches, and cluding finding, categorizing and logically structuring may even lead to ways of combining the advantages of legal information, and exposing it to the user of the sys- both approaches. tem in a way that is understandable to them. Here, we perform some initial investigations of whether ChatGPT, 4.1. Evaluation criteria a sophisticated general language model, can supply indi- Developing evaluation criteria ensures that we assess viduals with legal information, without specifically being ChatGPT’s performance in an objective, accurate and adapted to the legal domain. systematic manner. We drew upon the "HHH" (help
At the same time, it is important to remain aware of ful, honest, harmless) comparison criteria from previous the challenges of interacting with laypeople using AI. research[39]. Here, we believe that "helpfulness" not only Judges and lawyers, who have specialized training in law refers to ChatGPT’s ability to provide accurate and reschool, are better equipped to spot errors and flaws in liable information, but also encompasses its ability to the AI generated legal contents. However, lay people help users learn how to use the tool smoothly through a may not be able to evaluate the accuracy of the provided gentle learning curve. In addition, an important aspect information. So when we talk about legal self-help tools, of evaluating the helpfulness of ChatGPT is whether it legal information accuracy above the "pass mark" is not enough because legal self-help tools will serve a wide 5https://justicebot.ca can recommend some aspects that users may not have considered based on their specific situation. Therefore, we adapted the criteria to the objectives of this study. In the resulting evaluation comparison criteria, we aim to test two tools for: • Language comprehension. Should understand natural language and legal terms described by a layperson, while efectively communicating legal information to users in a clear and understandable way. • Accuracy. Should provide information and advice that is correct, reliable, and consistent with legal sources such as statutes, regulations, and case law. As the law is dynamic and subject to change, the information obtained by users from the tool should also be timely and up-to-date.[2] • Completeness. Should provide users with the necessary legal context and guidance on when and how to apply the provided legal information in diferent scenarios. Additionally, it should ofer concrete and actionable next steps for users to follow. • Trustworthiness. Should not provide misleading information, which includes wrong information and incomplete information. Even if the information provided is accurate, it may cause users to take wrong actions or lead to adverse consequences due to the lack of relevant context.
Trustworthiness includes more than just accuracy. A trustworthy tool not only provides accurate information but also ensures that users can confidently apply this information in the corresponding scenarios. Trust is the user’s perception, representing a consistent experience of receiving trustworthy information throughout their repeated interactions with the tool. • Harmless. Among others, should not generate toxic or ofensive statements and interact with users in a positive and inspiring manner whenever possible. Should not favor specific individuals, organizations or interests and should remain neutral. Should not encourage users to engage in illegal, dangerous, or potentially harmful activities. Should respect the privacy of users and protect their personal information. • User-friendly. Should be easy to use, with low
requirements for users. 4.2. Case Generation Everyday legal disputes, which are typically lowintensity but high-volume (compare [40]), are among the most common types of cases that the general public encounters. As described above, in these cases, the inability to obtain accurate legal information and advice can be a significant impediment to accessing justice. To evaluate how ChatGPT and JusticeBot can help in these situations, we generated cases that are representative of everyday legal issues. Since the public version of JusticeBot mainly covers landlord-tenant cases, in order to ensure the feasibility of comparison, we limit the selected cases mainly to this area.
To avoid potential bias that could result from the cases we used being included in ChatGPT’s training dataset, we did not choose to select real cases, but instead used ChatGPT for case generation, and then manually selected and adjusted cases to ensure that the selected cases cover as many aspects of legal information provision ability as possible.
The prompt we use is: "As a law professor, you need to develop simulated landlord-tenant cases for teaching purposes. Please provide three cases, detailing the parties involved, what happened, and who needs to seek legal information, and note that the cases occurred within the jurisdiction of Quebec."
Finally, the cases we use in this paper are: • Miller v. Johnson. In this case, Ms. Johnson rented a property to Mr. Miller. After Mr. Miller moved in, he was late in paying rent from the second month onward. At one year after moving in, Mr. Miller stopped paying rent altogether. Ms.
Johnson asked Miller to fix the problem, but he refused. Ms. Johnson then sought legal advice and sued Mr. Miller for breach of contract. This case took place in Quebec, Canada. • Jones v. Smith. In this case, Ms. Jones rented a property from Mr. Smith. After moving in, Ms. Jones discovered that the property had a serious pest infestation that made it unlivable.
She asked Mr. Smith to fix the problem, but he refused. Ms. Jones then withheld rent payments until the problem was resolved. Mr. Smith sued Ms. Jones for non-payment of rent. This case took place in Quebec, Canada. • Johnson v. Smith. In this case, Mr. Johnson rented an apartment from Mr. Smith. The lease agreement stated that pets were not allowed in the apartment. However, after moving in, Mr.
Johnson purchased a small dog and kept it in the apartment. Mr. Smith discovered the dog during a routine inspection and demanded that Mr. Johnson get rid of it. Mr. Johnson refused and argued that he had a legal right to keep the dog as a companion animal for his mental health.
Mr. Smith disagreed and threatened to evict Mr.
Johnson if he did not get rid of the dog. This case took place in Quebec, Canada.
As an aside, in this use ChatGPT proved to be a power- pathway, and users only need to make a few simple clicks ful tool for the generation of synthetic data. This could be to get the information they need, and the user experience an interesting use case in research where example cases is also very good. need to be evaluated, or used for training a machine Language comprehension. ChatGPT has a good unlearning model (see e.g. [41]). derstanding of natural language. Even with some typos and confusion in the description, it can still understand 4.3. Prompting process what the user means. ChatGPT also has the ability to understand diferent ways of describing the same or simUsers who utilize legal self-help tools may lack a legal ilar situations, such as bed bugs and pest infestation. In background and therefore have dificulty articulating JusticeBot, however, the paths that users can choose are their situation clearly or omitting key information. In limited by the scope of the system development. The user some cases, they may not even be sure what type of legal is asked to determine for themselves whether their situinformation they need. To test ChatGPT’s analytical abil- ation may fulfill certain legal criteria, and are provided ities in the face of specific cases, we use it to simulate a with context in the form of case law and plain language scenario where a litigant communicates with a lawyer. In descriptions to help them with this. the generated prompts, we include common errors and This language understanding capability of ChatGPT omitted key information to determine if ChatGPT can has significant implications for legal self-help tools, meanrecognize self-contradictory points in a user’s descrip- ing that lay people can more easily articulate their needs tion and complete missing information in subsequent and get answers when using these tools. conversations. The answers ChatGPT provides rarely contain complex
The prompt we use is "Suppose you are [party], you legal jargon, instead explaining the information users have no background knowledge of the law, and you are a need in plain English. However, It tends to generate exparty to the following case: [case fact]. Please simulated cessively long responses. While in some cases detailed what you would say as a lay person to your lawyer when responses can provide users with more relevant informayou are confronted." tion, in other cases ChatGPT generates responses that are
After obtaining simulated descriptions of parties that simply semantic repetitions of the same meaning, which could be provided to lawyers, we tested ChatGPT using may be confusing for the user. dialogue and followed up with further questions based Accuracy. The biggest shortcoming of using Chaton its responses to assess its ability to provide legal in- GPT to directly provide legal information is the lack of formation. accuracy of those answers. It frequently "hallucinates"
To avoid the interference of ChatGPT’s ability to re- answers to legal questions, generating false legal provimember consecutive conversations with the results, we sions and false cases. tested each case in a new session. Although JusticeBot ChatGPT provided correct information in some of our does not have the ability to remember the previous con- experiments, such when we tested the case Miller v. Johnversation, each test was also conducted in JusticeBot’s son (see 4.2), where ChatGPT correctly identified the new session in order to ensure the comparability of the relevant dispute resolution institution as the Tribunal adexperiment. ministratif du logement (formerly the Régie du logement). This may mean that ChatGPT has a higher accuracy in 5. Results providing non-numbered and more general information. In the Jones v. Smith Case, ChatGPT provided content Overall, ChatGPT and JusticeBot have diferent advan- that was quite close to the actual legal provisions, but the tages in diferent aspects. ChatGPT has very good perfor- information it provided deviated from the actual legal mance in language comprehension. However, its perfor- provisions in terms of key information. This could cause mance in information accuracy, completeness and trust- additional confusion to the user, as the provisions appear worthiness is somewhat lacking, while JusticeBot per- to be credible, relevant to the case, and the corresponding forms well in these aspects. Both tools perform well legal article numbers are quite close to the actual numin terms of harmlessness, with ChatGPT not generating bers. In the context of users without legal training, there harmful information during our experiments, and Justice- is a high probability that users will trust the information Bot avoiding it altogether due to its deterministic nature. provided by ChatGPT.
In terms of user experience, ChatGPT requires little adap- Surprisingly, in the Johnson v. Smith Case, ChatGPT tation due to its natural language interface, and users can provides legal content that is very similar to the original continuously follow up on the answers they receive, like Charter of human rights and freedoms. Although the they may do when speaking to a lawyer. JusticeBot uses content of the article is almost correct, it is not fully a series of simple questions to help users find the correct relevant to the case at hand.
Completeness. JusticeBot does a better job at providing complete information, and ChatGPT has shown the ness of ChatGPT in providing legal information. It is ability to proactively provide additional relevant infor- dificult for a layperson to point out errors in the inmation in some cases. formation provided, and ChatGPT does not reflect and
In the case of Jones vs. Smith, ChatGPT addressed admit that its answers were faulty in follow-up questions, not only the resolution of a housing infestation and the which can cause significant problems for the normal user. landlord’s obligations, but also whether the tenant had Since the information provided by JusticeBot is vetted the right to reduce or suspend rent payments. by experts, JusticeBot’s performance in terms of trust
JusticeBot exhibits a higher level of completeness in worthiness is more reassuring. providing legal information. For instance, users may Harmless. In our experiment, both ChatGPT and Jusoverlook important legal facts such as whether the other ticeBot used polite and professional language, without party has been notified or when the other party is overdue generating any toxic or harmful information. Likewise, on paying rent. It is often challenging for laypeople to we did not observe any noticeable bias favoring one side, actively provide such information when selecting a legal demonstrating their satisfactory performance in this repathway. JusticeBot overcomes this issue by providing gard. However, it should be noted that while OpenAI has relevant information during the user’s pathway guidance, implemented filtering mechanisms to prevent ChatGPT as well as ofering relevant legal cases for user reference. from generating harmful content, there is still a possibilThus, users are likely to have a higher probability of ity for the system to produce such content under certain getting comprehensive information during the use of coercive circumstances.
JusticeBot. In terms of user privacy and protection of personal
During our experiments, we observed that JusticeBot’s information, ChatGPT oefrs an option to not use the ability to provide helpful information to users is limited conversation data for further model training. As for Juswhen they encounter situations that are not covered by ticeBot, it does not collect any additional personal inforthe system. This is a deliberate tradeof to make sure that mation from users apart from the choices made based the information provided is relevant. However, it means on the questions presented in the legal pathway and that the JusticeBot system requires regular maintenance anonymized statistics. and expansion by experts to cover a broader range of legal User-friendly. Overall, both ChatGPT and JusticeBot matters. On the other hand, ChatGPT is highly scalable ofer excellent user interaction. ChatGPT enables direct and can answer nearly any question posed by the user, communication with users in natural language, reducbased on the enormous amount of data it has been trained ing the learning curve and making it easier to use. In on. Of course, this may be less of an advantage if the contrast, JusticeBot streamlines the process of obtaining information it provides is often wrong. legal information by allowing users to select the appropri
Trustworthiness. As mentioned above, ChatGPT ate options rather than having to describe their situation generates false information, especially about legal pro- in detail. visions and legal cases. ChatGPT also showed strong confidence in its generated content and was less willing to modify its answers when users repeatedly asked them 6. Discussion and Conclusion about it. For example, ChatGPT provided us with a case called Tan v. Bailey [2018 ONCA 260]. Despite our repeated confirmation, ChatGPT insisted that the case was genuine.
Prompt: are you sure it is a real case?
Answer: Yes, I can confirm that Tan v. Bailey [2018 ONCA 260] is a real case that was heard by the Ontario Court of Appeal in 2018. The case can be found on the CanLII website, which is a free online legal database in Canada. If you are interested in learning more about the case or reading the court’s decision, I would recommend searching for it on the CanLII website or another legal database.
However, through the search in CanLII, the real [2018 ONCA 260] case should be R. v. Jennings6.
This behavior substantially reduces the trustworthi
ities in diferent domains. ChatGPT ofers an outstanding interactive experience with minimal learning costs for users, allowing them to describe their legal matters using fragmented language and subsequently correct or reinforce the facts during the conversation. However, ChatGPT occasionally generates "hallucinations" in the legal field, an issue which may be addressed to some extent in GPT-4 [42]. As mentioned above, since legal information tools often target laypeople that are unable to verify the information provided, it is very important that the information is accurate, up-to-date, and sourced legal information. Given that ChatGPT is a language model rather than a knowledge database, it does not generate information with perfect accuracy. OpenAI seems to be well aware of this limitation, and specifically prohibits the use of its language models to provide specific legal
retrieved on 2023-03-15 advice.7 [5] H. Westermann, K. Benyekhlef, Justicebot: A
JusticeBot, on the other hand, shines in its ability to methodology for building augmented intelligence provide accurate and deterministic legal information, ver- tools for laypeople to increase access to justice, in: ified by legal experts. Of course, inputting this informa- Proceedings of the Nineteenth International Contion for new legal areas can take time, which can make ference on Artificial Intelligence and Law, 2023. it dificult to cover every legal domain a user may face. [6] L. K. Branting, Advisory systems for pro se litigants,
Given the diferent tradeofs between the approaches in: Proceedings of the 8th international conference of JusticeBot and ChatGPT, an interesting approach could on Artificial intelligence and law, 2001, pp. 139–146. be combining the two. Tools such as the JusticeBot could [7] J. Zeleznikow, Using web-based legal decision supbe used to inject verified and accurate knowledge to Chat- port systems to improve access to justice, InformaGPT. For example, ChatGPT could be used as the com- tion & Communications Technology Law 11 (2002) munications layer (compare [8]) that communicates with 15–33. the user and makes the information accessible to them [8] L.-C. Paquin, F. Blanchard, C. Thomasset, Loge– (compare [43]). ChatGPT could also guide the user to- expert: from a legal expert system to an information ward the correct legal pathway in the JusticeBot, thereby system for non-lawyers, in: ICAIL 1991, 1991, pp. helping them understand which pathway is relevant for 254–259. their situation. We previously explored such an approach [9] H. Westermann, V. R. Walker, K. D. Ashley, in [41], using other language models. We will continue K. Benyekhlef, Using factors to predict and aninvestigations in this direction in future work. alyze landlord-tenant decisions to increase access
Overall, as we can see, ChatGPT is not yet accurate to justice, in: Proceedings of the Seventeenth Interenough to provide legal information directly to laypeople. national Conference on Artificial Intelligence and ChatGPT and JusticeBot have diferent strengths in pro- Law, 2019, pp. 133–142. viding legal information — a very promising avenue may [10] M. Bommarito II, D. M. Katz, Gpt takes the bar be the combination of the two approaches, to create a exam, arXiv preprint arXiv:2212.14402 (2022). powerful tool that provides comprehensive legal support. [11] OpenAI, Gpt-4, 2023. URL: https://openai.com/ We are looking forward to exploring such approaches in research/gpt-4. future work. [12] D. M. Katz, M. J. Bommarito, S. Gao, P. Arredondo, Gpt-4 passes the bar exam, Available at SSRN 4389233 (2023).
Acknowledgments [13] J. H. Choi, K. E. Hickman, A. Monahan, D. Schwarcz, Chatgpt goes to law school, Available at SSRN Jinzhe Tan, Hannes Westermann, and Karim Benyek- (2023). lef would like to thank the Cyberjustice Laboratory at [14] A. M. Perlman, et al., The implications of openai’s Université de Montréal, the LexUM Chair on Legal Infor- assistant for legal services and society, Available at mation and the Autonomy through Cyberjustice Tech- SSRN (2022). nologies (ACT) project for their support of this research. [15] J. Savelka, K. Ashley, M. Gray, H. Westermann, H. Xu, Can gpt-4 support analysis of textual data References in tasks requiring highly specialized domain expertise?, in: ASAIL’23: 6th Workshop on Automated [1] OpenAI, Introducing chatgpt and whisper Semantic Analysis of Information in Legal Text, apis, 2023. URL: https://openai.com/blog/ 2023.
introducing-chatgpt-and-whisper-apis/. [16] J. Savelka, Unlocking practical applications in le[2] M. Lauritsen, Q. Steenhuis, Substantive legal soft- gal domain: Evaluation of gpt for zero-shot seware quality: A gathering storm?, in: Proceedings mantic annotation of legal texts, arXiv preprint of the Seventeenth International Conference on Ar- arXiv:2305.04417 (2023).
tificial Intelligence and Law, 2019, pp. 52–62. [17] J. Savelka, K. Ashley, M. Gray, H. Westermann, [3] M. Galanter, Access to justice in a world of expand- H. Xu, Explaining legal concepts with augmented ing social capability, Fordham Urb. LJ 37 (2010) large language models (gpt-4), in: AI4Legs 2023: AI 115. for Legislation, 2023. [4] M. Cappelletti, B. Garth, Access to justice: the [18] H. Westermann, J. Savelka, K. Benyekhlef, Llmedinewest wave in the worldwide movement to make ator: Gpt-4 assisted online dispute resolution, in: rights efective, Buf. L. Rev. 27 (1977) 181. Artificial Intelligence for Access to Justice (AI4AJ 2023), 2023. [19] J. Wei, Y. Tay, R. Bommasani, C. Rafel, B. Zoph, 7https://openai.com/policies/usage-policies S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large lan- nowhere to go’ self-help legal aid services in ausguage models, arXiv preprint arXiv:2206.07682 tralia, Alternative Law Journal 26 (2001) 184–190. (2022). [37] J. E. Cabral, A. Chavan, T. M. Clarke, J. Grea[20] C. Zhen, Y. Shang, X. Liu, Y. Li, Y. Chen, D. Zhang, cen, Using technology to enhance access to justice, A survey on knowledge-enhanced pre-trained lan- Harvard Journal of Law & Technology 26 (2012) guage models, arXiv preprint arXiv:2212.13428 241–324.
(2022). [38] J. Bennett, T. Miller, J. Webb, R. Bosua, A. Lodders, [21] F. Yu, L. Quartey, F. Schilder, Legal prompting: S. Chamberlain, Current state of automated legal Teaching a language model to think like a lawyer, advice tools (2018).
arXiv preprint arXiv:2212.01326 (2022). [39] A. Askell, Y. Bai, A. Chen, D. Drain, D. Gan[22] D. Trautmann, A. Petrova, F. Schilder, Legal prompt guli, T. Henighan, A. Jones, N. Joseph, B. Mann, engineering for multilingual legal judgement pre- N. DasSarma, et al., A general language assisdiction, arXiv preprint arXiv:2212.02199 (2022). tant as a laboratory for alignment, arXiv preprint [23] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, arXiv:2112.00861 (2021).
Q. Le, D. Zhou, Chain of thought prompting elicits [40] K. Benyekhlef, J. Zhu, Intelligence artificielle et reasoning in large language models, arXiv preprint justice: justice prédictive, conflits de basse intensité arXiv:2201.11903 (2022). et données massives, Intelligence 30 (2018). [24] N. Guha, D. E. Ho, J. Nyarko, C. Ré, Legalbench: [41] H. Westermann, S. Meeùs, M. Godet, A. Troussel, Prototyping a collaborative benchmark for legal J. Tan, J. Savelka, K. Benyekhlef, Bridging the gap: reasoning, arXiv preprint arXiv:2209.06120 (2022). Mapping layperson narratives to legal issues with [25] B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, language models, in: ASAIL’23: 6th Workshop on J. Yue, Y. Wu, How close is chatgpt to human ex- Automated Semantic Analysis of Information in perts? comparison corpus, evaluation, and detec- Legal Text, 2023.
tion, arXiv preprint arXiv:2301.07597 (2023). [42] OpenAI, Gpt-4 technical report, 2023. [26] B. S. Noveck, Wikipedia and the future of legal arXiv:2303.08774.
education, J. Legal Educ. 57 (2007) 3. [43] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, [27] A. Currie, The legal problems of everyday life, M. Lomeli, L. Zettlemoyer, N. Cancedda, T. Scialom, in: Access to justice, Emerald Group Publishing Toolformer: Language models can teach themselves Limited, 2009. to use tools, arXiv preprint arXiv:2302.04761 (2023). [28] T. C. Farrow, A. Currie, N. Aylwin, L. Jacobs,
D. Northrup, L. Moore, Everyday legal problems and the cost of justice in canada: Overview report, A. Support materials
Osgoode Legal Studies Research Paper (2016). [29] L. Savage, S. McDonald, Experiences of serious Support materials are stored at https://github.com/ problems or disputes in the canadian provinces, JinzheTan/ChatGPT-as-an-Artificial-Lawyer- The file 2021, Juristat: Canadian Centre for Justice Statistics “ChatGPT as an Artificial Lawyer appendix.docx” has (2022) 1–28. the original transcript of the conversation with ChatGPT [30] J. Macfarlane, The national self-represented liti- and JusticeBot.
gants project: Identifying and meeting the needs of self-represented litigants final report (2013). [31] K. Branting, C. Balhana, C. Pfeifer, J. S. Aberdeen,
B. Brown, Judges are from mars, pro se litigants are from venus: Predicting decisions from lay text., in: JURIX, 2020, pp. 215–218. [32] K. A. Sabbeth, Housing defense as the new gideon,
Harv. Women’s LJ 41 (2018) 55. [33] L. Sudeall, The overreach of limits on" legal advice",
Yale LJF 131 (2021) 637. [34] M. Galanter, Why the haves come out ahead: Speculations on the limits of legal change, Law & Soc’y
Rev. 9 (1974) 95. [35] L. S. o. U. Canada, Access to Justice for a New Century: The Way Forward, Law Society of Upper
Canada, 2005. Google-Books-ID: zxN9QgAACAAJ. [36] J. Giddings, M. Robertson, ‘informed litigants with