A variety of computational tools and systems have become available in recent years to assist pro se (self-represented) litigants in formulating claims and defenses in legal forums. However, there has been little exploration of techniques for evaluation of these systems to establish their relative efectiveness and identify specific strengths and weaknesses of each approach. This paper proposes the Justice Access Game, a crowd-sourced method for comparative evaluation of legal information systems. The Justice Access Game is a framework applicable to wide range of domains and systems.
evaluation [6]. As a result, it is dificult to measure the
performance of alternative technical approaches in terms
Worldwide, a significant proportion of litigants appear either of accuracy, usability, efectiveness, or success rate.
pro se, i.e., without the assistance of an attorney, in courts Instead, relative performance is typical estimated
anecand other decision forums [
Development of approaches to assisting pro se litigants dress the challenges of evaluating legal assistant systems.
is an area of active research and exploration. One ap- The next section describes the design of the JAG,
Secproach has been to replace conventional decision bodies, tion 3 sets forth the pragmatics of the game, the possible
which are often extremely challenging for non-experts to role of Large Language Models is discussed in Section 4,
understand and navigate, with alternative dispute resolu- and a summary and proposal for future work is set forth
tion forums, such as ODR platforms, which are designed in Section 5.
to be less disadvantageous to pro se litigants [
Currently, the most common alternatives to simple Players compete to identify a legal claim, identify all
online form and instruction resources automated legal facts supporting the claim, and assert the claim via a
leassistant systems consist of some form of guided form- gal filing. Points are awarded for achieving these goals
iflling software, sometimes implemented as a chatbot and for incorporating all relevant but no irrelevant facts.
[12], in computable logic [14], or combining these ele- Points are reduced based on errors and the duration or
ments with predictive models [17]. In the United States number of conversational turns required to achieve the
and Canada, many such systems have been developed goal. While individual participants compete for points,
through a collective process involving communities of alternative systems are compared via the average number
attorneys and other public-interest advocates [16] [7] of points that competitors are able to achieve using those
[
After being provided with a factual scenario and assigned a role and a goal, the player must accomplish the following tasks using the system: 1. Determine what legal action would achieve the
goal, if successful 2. Identify all facts necessary for each element of
the legal action 3. Draft a claim or petition setting forth the claim
and all elements needed to support the claim At the conclusion of each round, the player is assigned a score based on a weighted combination of the following factors: 1. Is the legal action correct, i.e., if the claim suc
ceeded, would it achieve the player’s goal? 2. Were the elements required for the action identi
ifed? 3. Were all the facts relevant to those elements iden
tified? 4. Were there extraneous facts? 5. How many steps or how long did the process
take?
Systems are ranked based on the average number of points that competitors are able to achieve using each system.
Consider an example game. Suppose that the factual scenario were as follows:
Under this set of facts, a player could be assigned the role of Mary, John, or Bill. Each of these roles is associated with diferent set of possible legal goals, as shown in Table 1.
Mary John Bill
Keep John out of their apartment Keep John from being alone with Nell.
Prevent eviction Return to apartment.
Spend time alone with Nell.
Prevent eviction, if he is allowed to return to apartment Evict John and Nell.
Keep John away from Bill
For example, if a player were given a certain system, assigned the role of Mary, and given Mary’s Goal 1 (keeping John out of the apartment), the player’s performance using the system would be scored based on the the following: • Identifying that Goal 1 could be achieved by a
protection order • Listing requirements for a protection order: – Domestic relation (cohabiting and having
a child together) – Actual or threatened harm (injury to Nell) • Completion of all fields of a petition for a protec
tion order as Mary
Facts John and Mary started dating in September 2017, and in January 2018 they moved into an apartment that they A fact pattern, together with a single role and a sinrented from Bill. Baby Nell was born in 2019. Starting in gle goal constitutes a problem. A fact pattern together 2021 the radiator in the apartment began failing during with the sets of roles and goals arising from that fact cold snaps. John and Mary complained about the cold, pattern, constitutes a problem set. The example above and without telling Mary, John stopped paying rent in illustrates that a single problem set may include problems February 2021 to protest the condition of the apartment. in multiple areas of law.
When John and Mary received an eviction notice in March, Mary became very angry at John. In the ensuing argument, a window in the apartment was broken and 2.3. Scoring Nell received a cut on the forehead, which Mary blamed on John. When Bill came down to investigate the noise and saw the broken window, John threatened to hurt Bill unless he fixed the heat. Bill left the apartment and called the police, but John left before the arrival of the police, who listened to the arguments and left without taking any action. Mary refuses to let John in the apartment or see Nell. Bill insists that Mary leave the apartment immediately. John refuses to pay the rent until Bill fixes the radiator and insists that he has a right to live in the apartment with Mary and to spend time alone with his daughter.
As described above, the scoring of each individual problem is based on correctness, completeness, accuracy, and duration. The Justice Access Game does not depend on any particular procedure for performing this scoring. The most informative scoring would be individual ratings by experts in the particular areas of law involved in the problem set, e.g., legal-aid attorneys or law professors. A more practical alternative would be a heuristic match to model solutions, e.g., ROUGE scores [8] or similarity in semantic-embedding space [10] between each solution element and the corresponding element of the model solution.
3.1. Game Requirements and Preparation Applying the Justice Access Game requires that the systems to be compared share a common legal domain. For example, a chatbot could be compared with a guidedpathway system only if both were designed to address a common legal problem.
A more challenging pragmatic issue is that a model solution is needed for each problem. Justice Access Game problems are intended to be representative of the challenges routinely faced by pro se litigators, so the solutions should be straightforward for legal-aid attorneys or law professors. However, preparing all the solutions for a entire problem set could require an investment of many hours.
Participation in the Justice Access Game is intended to approximate the experience of actual pro se litigants, who typically have little or no experience with judicial proceedings. Appropriate participants could therefore include anyone with limited legal experience, e.g., Mechanical Turk crowd-workers, participants in the NSF Timesharing Experiment,1 or college students. By contrast, law students would probably not be appropriate. For the same reason, players should not be given multiple problems involving the same area of law.
The key objective of the Justice Access Game is to empirically establish the efectiveness of alternative paradigms for assisting pro se litigants. Accordingly, it is appropriate to start with the most widely used baselines. The ifrst would consists of court forms plus the Internet, the resources traditionally available to pro se litigants in a typical court. A second proposed baseline would consist of the online form technologies currently promoted by Legal Services Corporation, e.g., Court Forms Online2 or Maryland Court Help.3
The capabilities of Large Language Models (LLMs), such as ChatGPT [13] have recently advanced dramatically. When JAG problems are posed to these systems, they are capable of providing advice that is mostly useful and accurate, but occasionally disastrously incorrect. In an 1https://www.tessexperiments.org/ 2https://courtformsonline.org/ 3https://perma.cc/F654-KMNE example session, when ChatGPT (3.5) was given the example fact pattern from Subsection 2.2 above and was asked what advice it would give if it were a legal aid attorney, it provided the following response:
Mary can take legal action to keep John out of the apartment by obtaining a protective order or a restraining order.
A protective order is a court order that prohibits an abuser from having contact with the victim and may require the abuser to stay away from the victim’s home or workplace. In Virginia, a victim of domestic violence, such as Mary, can apply for a protective order from the court.
To obtain a protective order, Mary would need to file a petition with the circuit court in the county where she lives or where the abuse occurred. She would need to provide detailed information about the abuse, including dates and specific incidents, and explain why she needs protection. She should also provide any evidence she has, such as medical records, photographs, or witness statements, to support her allegations.
If the court finds that Mary is in danger of further abuse, it may grant a protective order that prohibits John from having any contact with her or coming within a certain distance of her and her daughter.
Violating a protective order is a criminal ofense and can result in John being arrested.
Alternatively, Mary can also seek a restraining order, which is a court order that prohibits someone from coming near or contacting the protected person. In Virginia, a restraining order can be obtained through the Juvenile and Domestic Relations District Court.
Mary should consider consulting with a local attorney who can help her assess her options and determine the best course of action for keeping John out of the apartment and protecting herself and her daughter.
When asked to draft a sample protection order, the result was the following:
As an AI language model, I’m not authorized to provide a completed legal document as it requires legal expertise and access to specific information about the case. However, I can provide an example of the information that Mary may include in a Virginia petition for a restraining order based on the facts provided: [Header information: Virginia Circuit Court, [Name of County], Civil Case No. [insert case number]] PETITION FOR A PROTECTIVE ORDER My name is [Mary], and I reside at [address]. The person I am seeking protection from is [John], and his last known address is [address]. John and I were in a relationship from September 2017 to March 2021. We have a daughter together, named [Nell], who was born in 2019. In February 2021, John stopped paying rent on our apartment to protest the condition of the apartment. Despite my requests to pay the rent, John refused to do so, resulting in us receiving an eviction notice in March. On [date], John and I had an argument about the eviction notice, during which a window in the apartment was broken, and Nell received a cut on her forehead. I believe that John broke the window and caused the injury to our daughter. When our landlord, Bill, came down to investigate the noise and saw the broken window, John threatened to hurt Bill unless he fixed the heat. Bill left the apartment and called the police, but John left before their arrival. I am in immediate and present danger of family abuse by John, and I am seeking a protective order to ensure my safety and the safety of my daughter. I request that the court grant a protective order that prohibits John from having any contact with me or my daughter and from coming within a certain distance of us. [Signature of petitioner, Mary] Again, this is just an example of the type of information that Mary may include in a petition for a restraining order in Virginia. It is important to note that legal documents such as this require a high degree of legal expertise and accuracy, and it is recommended that Mary consult with a local attorney for assistance in drafting and filing the petition.
It seems almost certain that we stand on the threshold of an era in which pro se litigants will increasingly turn to LLMs for legal advice, notwithstanding any disclaimers such as those appearing above, against providing legal advice. The Justice Access Game may provide one mechanism for assessing the quality and usefulness of this advice to pro se litigants.
This paper has described a new crowd-sourced method for comparative evaluation of legal information systems, the Justice Access Game. The Justice Access Game provides a framework applicable to wide range of domains and systems and can provide a realistic assessment of the capabilities of systems to assist pro se systems in a realistic setting. This paper has briefly set forth the details of the Justice Access Game and provided two example games. It is hoped that this work will be of value in understanding the capabilities of emerging technologies, including not just extensively engineered solutions such as the JusticeBot [17] but also rapidly changing ad hoc capabilities typified by ChatGPT [13].
I gratefully acknowledge the assistance of Treyvon Jordan and Stephen Steward in developing Justice Access Game scenarios. The MITRE Corporation is a not-forprofit company, chartered in the public interest. This document is approved for Public Release; Distribution Unlimited. Case Number 23-1584. ©2023 The MITRE Corporation. All rights reserved.
A. Justice Access Game Example 2 Psychology, Public Policy, and Law 26(2), 198–212 (2020) [6] Lauritsen, M., Steenhuis, Q.: Substantive legal soft- Facts ware quality: A gathering storm? In: Proceedings Tracy let an ex-boyfriend, Mark, borrow her car after of the Seventeenth International Conference on Ar- many assurances that it would be a quick trip “down the tificial Intelligence and Law. pp. 52–62. ICAIL ’19, block”. An hour later, Tracy received a call from Mark ACM, New York, NY, USA (2019) that he was pulled over by the police under a suspended [7] LawHelp Interactive (2023), https:// driver’s license. The police advised Mark that he was lawhelpinteractive.org/, last visited 15 April traveling through a known drug corridor as the reason 2023 for the stop. Mark was not aware that his license was sus[8] Lin, C.Y.: ROUGE: A package for automatic eval- pended prior to borrowing the car. Mark called the DMV uation of summaries. In: Text Summarization and was informed that his license was suspended due Branches Out. pp. 74–81. Association for Compu- to the lack of insurance coverage on his vehicle. Mark tational Linguistics, Barcelona, Spain (Jul 2004), had switched insurance providers recently, and believed https://aclanthology.org/W04-1013 this is the reason for the error. Mark told Tracy that the [9] Macfarlane, J.: The national self-represented liti- vehicle was impounded by the police department and gants project: Identifying and meeting the needs of was told that it was a 30-day impoundment and would self-represented litigants: Final report. CanLIIDocs cost $300 for release. Tracy was stunned because she p. 493 (2013) only had one vehicle and not having one will substan[10] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efi- tially afect her ability to work as an Uber driver. Tracy cient estimation of word representations in vector had recently moved into her current apartment and after space. CoRR abs/1301.3781 (2013), http://arxiv.org/ incurring expenses for new furniture, a security deposit, abs/1301.3781 and other costs associated with moving, a $300 loss would [11] Moore, S., Nwebury, A.: Legal aid in crisis: Assess- have a significant impact on her family’s finances. James, ing the impact of reform. Bristol University Press, Tracy’s husband (who thinks they overpaid on moving ex1st edn. (2017), https://doi.org/10.2307/j.ctt1t8988q penses and furniture) became angry after hearing about [12] Morgan, J., Paiement, A., Seisenberger, M., the additional $300 expense for the vehicle’s return. AfWilliams, J., Wyner, A.: A chatbot framework for ter the argument, James asked for a divorce, took their the children’s legal centre. In: Palmirani, M. (ed.) three-year-old son, and left the apartment. Legal Knowledge and Information Systems - JURIX 2018. Frontiers in Artificial Intelligence and
Applications, vol. 313, pp. 205–209. IOS Press (2018) [13] chatGPT, https://platform.openai.com/ [14] Pyle, J.: Docassemble (2019), https://docassemble.
org/ last accessed on APril 19, 2023 [15] Schmitz, A., Zeleznikow, J.: Intelligent legal tech to empower self-represented litigants. Columbia Science and Technology Law Review pp. 142–190 (March 2 2022) [16] Steenhuis, Q., Colarusso, D.: Digital curb cuts: Towards an inclusive open forms ecosystem. Akron
Law Review (August 25, 2021) [17] Westermann, H., Walker, V., Ashley, K., Benyekhlef,
K.: Using factors to predict and analyze landlordtenant decisions to increase access to justice.
In: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law. p. 133–142. ICAIL ’19, Association for Computing Machinery, New York, NY, USA (2019) Tracy James James
Goal Retrieve the vehicle before the 30-day deadline Avoid paying for the vehicle’s return or be compensated for doing so Divorce Tracy Obtain Joint-Legal Custody of Son If DC-499 is too lengthy a process, pay for the vehicles return, and sue Mark for the losses. This can be done in small claims court by filing a DC-402.
If no fault, submit a VS-4 Report of Divorce or Annulment Either the required forms in J&DR Court or file complaint from scratch to be included with the divorce in Circuit Court Provide the DMV with a Certification of Insurance Pay the DMC reinstatement fees Submit a Petition for Proceeding in Civil Case Without Payment of Fees or Costs to the General District Court
Prove you did not know the ofender’s license was suspended.
Prove that the immediate family only has one vehicle and will sufer hardship if not returned Demonstrate that Mark is the cause of Tracy’s injury (the costs incurred by the vehicle’s impoundment). Having been separated for at least a year Plead complaint in alignment with the “information considered in custody proceedings.” Provide the DMC with a Certification of Insurance Pay the DMC reinstatement fees Provide net income, assets, and expenses to the court.