Individuals without legal training (i.e., laypeople) typically tend to perceive their situation through facts, i.e., events that occur. Understanding which legal opportunities or remedies are available to them requires an analysis of which legal issues are raised by these facts, which may be dificult for laypeople to assess. This “gap” can cause laypeople to miss out on benefits or be unable to resolve their disputes. In this paper, we propose an approach to automatically analyze a factual description provided by a layperson in order to map it to potentially relevant legal issues. The system then suggests the issues to the user who may decide if and how to explore them. We demonstrate how this approach could be integrated in legal decision support tools, such as the JusticeBot, to guide users to the relevant guided pathways, while giving the user the possibility to verify the results. This has the potential to further increase the impact on access to justice of such tools. We evaluated the approach on real-world data collected in the JusticeBot project, and found that the system was able to identify the relevant legal issue in 93.5% of selected cases. Our findings can be leveraged by legal professionals and developers of legal decision support systems to alleviate the challenges related to bridging the gap between layperson language and legal issues.
the law provides [5].
A significant issue in providing legal information to laypeople is the gap between layperson language and legal language [6]. Laypeople often tend to think of their situation in terms of what has happened (such as “There relief, however, the layperson needs to link this factual occurrence to legal issues that can lead to remedies. Each factual situation may give rise to diferent legal criteria being fulfilled, and thus diferent remedies.
a failure of a landlord to fulfill their duties. Laypeople, however, may not be able to establish this link, which could lead them to struggle to understand which rights they have, or even that they have any legal rights at all. Knowing there is a legal right is, of course, an important precondition to commencing the enforcement of these rights. A survey conducted in 2009 showed that many of the individuals that did not act at all in response to a legal problem were not aware that their problem had a legal solution [2]. Further, laypeople may struggle to know which forms to employ when filing a claim in court [ 7], or which facts are relevant and need to be proved when
putes. Most laypeople (i.e. individuals without legal training) will face a legal dispute at some point in their lives. These may include, e.g., issues related to debts, employment or consumer rights [1]. Many individuals do not know how to efectively resolve such disputes.
veys show that many individuals believe they could have received a better result with more legal information [2].
Only a minority of people use the court system to resolve their issues. Using the court can be expensive, leading many individuals to self-represent. It can also nEvelop-O legal disputes that had come up over the past few years Proceedings of the Sixth Workshop on Automated Semantic Analysis of ation and a legal understanding of a situation can further afect the usefulness of legal self-help tools [ 6], a powerful way of increasing access to justice [9, 10, 11]. We have noticed this in the user-feedback of the JusticeBot, a legal decision support tool focused on landlord-tenant disputes, built at the Cyberjustice Laboratory. The system can ask questions of the user, analyze their responses, and then provide them with legal information regarding 2. Related and Prior work their situation and the potential next steps that they could consider undertaking to resolve their dispute. The first Westermann et al. describe foundational principles of the version of the JusticeBot, built in collaboration with the JusticeBot design and reports initial proof of concept extribunal administratif du logement (Housing tribunal of periments [13]. In a follow-up publication, Westermann Quebec), has been used by over 20,000 users (over 140k and Benyekhlef proposed a generalized methodology for page views). For a more complete description, see [12]. building augmented intelligence tools for laypeople to
After accessing the JusticeBot at https://justicebot.ca, increase access to justice [12]. Here, we devise a method users first select whether they are a landlord or a tenant. to extend this methodology, by suggesting relevant pathIf they choose, e.g., the tenant option, they are given a ways to the user based on layperson descriptions of faclist of legal issues that the system can handle. Figure 1 tual situations. shows the issues available for the tenant, such as “There The plain language movement1 has criticized the verare bedbugs in my apartment”, “I want to terminate my bosity of legal language, complicated syntax employed lease” or “Other”. in legal texts, as well as overuse of specialized terms. In
By clicking any of these options, the user will be taken the past, there have been studies exploring the possithrough a legal guided pathway that helps them assess bilities, efects and limitations of communicating legal their rights and understand the potential next steps they information in a way that is more accessible to laypeocan undertake to address their situation. If none of the ple [14, 15, 16]. There have been early attempts to creissues are relevant to the user, they can click the “Other” ate a legal information retrieval system for laypeople option, which will inform them that their issue is not yet [17]. Garimella et al. experimented with general natural covered by the JusticeBot. They further have the option language processing (NLP) text simplification methods to submit a form describing their situation, so that we on legal documents [18]. Uijttenbroek et al. describe can evaluate which pathways should be added to the a system that analyzes laypeoples input in terms of a JusticeBot. layperson ontology, maps the entities to a legal ontol
When analyzing the submitted descriptions, we noticed that a significant number of the issues described by the users were, in fact, covered by existing pathways
plainlanguage.gov/resources/articles/beyond-a-movement/ [Accessed 2023-5-3] ogy, retrieves relevant case law, and finally presents the results to the layperson in a comprehensible way [19]. Fernández-Barrera and Casanovas focused on mapping layperson queries to ontologies in the domain of consumer mediation [20]. [6] explored the diference between layperson language and judicial language, finding that layperson submissions are dificult to use to predict case outcomes. Spot is an API that can analyze nonlawyer descriptions and link it to a standardized list of legal issues.2 [21] compared the JusticeBot approach to asking ChatGPT questions in layperson language, finding that the answers given by ChatGPT had some issues when it comes to accuracy and reliability. Here, we propose an approach to map layperson factual descriptions to legal issues using language models, describe its use in the context of a decision support system, and evaluate its performance on real-world data.
In NLP, the success of word embeddings (e.g. [22, 23]) was followed by an increasing interest in learning continuous vector representations of longer linguistic units such as sentences. This trend that has been reflected in AI & Law research as well [24, 25, 26]. Cer et al. [27] utilized the transformer architecture [28] and Deep Averaging Network [29] trained on the SNLI dataset. Reimers et al. build on top of BERT [30] and RoBERTa [31], which have been shown to be remarkably efective on a number of NLP tasks. Specifically, they used siamese and triplet network structures to derive semantically meaningful sentence embeddings [32]. Conneau et al. demonstrated the efectiveness of models trained on a natural language inference task (SNLI dataset [33]). They proposed a BiLSTM network with max pooling trained with fastText word embeddings [34, 35] as the best universal sentence encoding method [36]. While most of the earlier work was limited to a one or few languages, several approaches to obtain general-purpose massively multi-lingual sentence representations were proposed [37, 38, 39]. Such representations were utilized in many downstream applications, such as document classification [ 40], machine translation [41], question answering [42], hate speech detection [43], or information retrieval (IR) in the legal domain [44]. In this work, we utilize a multilingual sentence encoder [39] to embed and compare factual descriptions written by laypeople.
how our method, linking layperson factual narratives to legal issues, can be used to support users of the JusticeBot. As discussed, users of the system frequently
struggle to identify the pathway that is relevant to their situation. Hence, we implemented a feature to automatically suggest a potentially relevant pathway based on a description of an issue provided by the user.
Figure 2 shows the new interface related to the new feature in the JusticeBot. Instead of just a list of possible pathway options (as can be seen in Figure 1), users are now also shown a text box, and are invited to describe their factual situation. While they are typing, the system will retrieve suggestions of relevant pathways and display them to the user. Figure 2 shows the result of entering “I am cold” into the text box. As we can see, the system suggests three possible pathways that may be relevant to the user. Each suggestion consists of the following elements: • Factual explanation: An explanation of what the system understood from the user’s description. This can help the user verify that the system has correctly understood their situation. In our example, the first entry in the list states: “You may have issues with heating or insulation.”, based on the factual situation described by the user. • Suggested action: An explanation of what will be accomplished by clicking the link , i.e. the legal issue the user may want to explore. For example, the first entry in Figure 2 indicates that the user may wish to explore whether their landlord has not fulfilled their duties, and the consequences of such a situation. The suggested action is also important where the same factual situation (e.g. heating not working) can lead to diferent legal remedies becoming relevant (e.g. rent reduction and/or lease termination). • Link: Once users click the suggestion, they will be taken: – To the relevant pathway, if the issue is covered by the JusticeBot. It can also take the user to specific locations in the pathway, the JusticeBot. These pathways describe diferent legal if the answers to certain questions are al- issues that a user may wish to explore. For each of these ready evident from the factual description pathways, we formulated multiple factual situations that of the user. could give rise to this pathway being relevant for a user. – To an external site that has more informa- Then, we put ourselves in the shoes of a user that faces tion about the legal situation, if the issue this factual issue, and imagined how a user may describe is not covered. While providing the user their situation, in their own words. For each suggestion, with verified information in the JusticeBot we thus end up with the following elements: issspouopnrrcedefiesnrgcaabJnules,btilecinehBkeiolnptgfpuatlothwtwrhuaesyrteehdathseexntoceotrrynreaetl- 1. sAonnummabyeerxopfreexsas mthpelier dseitsucartipiotnio.nFsohroewxaamlpaylep:erbeen created.
As we can see, just like the JusticeBot itself, the pathway suggestion system acts as an augmented intelligence system, by suggesting pathways or external sources to the user, but never telling them what to do. By reading the factual explanation, the users are able to verify that the system has correctly understood their situation. We will explore this feature more in depth in Section 6.
Next, let us take a look at the technical stack that enables the functionality of finding relevant pathways based on a layperson’s factual description of a situation. The key idea underlying the approach is that the user query is not compared directly to the legal issue, but rather to a database of example descriptions of situations that would be covered by a certain legal issue.
The process consists of four steps: • Section 3.3 - The creation of example descriptions of situations that would lead to certain legal issues becoming relevant. • Section 3.4 - The creation of sentence embeddings for each example description. • Section 3.5 - The indexing of these sentence embeddings, in order to be able to quickly retrieve example descriptions similar to a new description. • Section 3.6 - In order to suggest relevant pathways to the user, we create an embedding of the factual description provided by the user and use the index to retrieve similar example descriptions, that are then used to suggest the relevant legal issue.
3.3. Creation of example situations more likely it is that the description of the user will be similar to a previously created description (see section The first step is the creation of a database of content that 3.6). To develop the database of example user descripcan be used to match the user’s description of a situa- tions, we use two sources: tion. We started by creating such descriptions ourselves, Seed example descriptions: As described above, we considering the legal issues that are currently covered by write our own seed examples that describe how a user • I need to wear a jacket indoors. • When I wake up, I have bites on my face. • I would like to go on vacation, what can I
do? • I received a letter from my landlord, in
forming me of a rent increase.
fying what the model understood from the user description (e.g. “You may have heating issues”, “You may have a bedbug infestation” or “You seem to want to leave your apartment for a while”, see Section 3.1) 3. A suggested action that the user will undertake by pressing the suggestion (e.g., “Explore whether the landlord has breached their obligation to keep the apartment warm”, or “Read more about bedbugs on an external site”, see Section 3.1) 4. A link to the relevant legal issue, as covered in the JusticeBot. This can be the beginning of a pathway, or a deep link into the pathway. It can also be a link to external sources.
For example, linking the situation of a tenant being cold in their apartment to a pathway exploring whether the tenant can receive a rent reduction, these elements would be as follows: 1. Example descriptions of heating issues: • I need to wear a jacket indoors. • There is frost on the inside of my window. • I feel cold indoors every day • etc. 2. “You seem to have heating issues” 3. “Explore whether you can receive a rent reduction due to the landlord not fulfilling their obligations.” 4. A reference to a relevant JusticeBot pathway. might express their situation. We tried to be as varied as possible in formulating these prompts, to be able to provide relevant suggestions for as many user descriptions as possible. Many diferent types of descriptions can lead to the same pathway becoming relevant. For example, both “I would like to sublet my apartment” and “I am going on vacation” and “My friend wants to live in my apartment for a few months” could guide the user to explore the legal situation of subletting. Due to the power of language models, as we will see, it does not matter if the way the user describes their situation perfectly matches the example formulation—as long as the meaning is similar, the suggestion that is the most similar will often be correct.
User-submitted example descriptions: Second, as described in section 1, users of the JusticeBot tool are invited to submit a description of their situation if their case is not covered by the JusticeBot. These are yet another high-quality source of data, since they represent the real-world users’ descriptions of their situations. Thus, by using these descriptions, we are able to match the way laypeople describe their situations.
Beyond adding the user-submitted examples as training data to benefit the matching methodology, they can also be used to evaluate the performance of how well the suggestion feature works. Each submitted missing question description represents a user that was not able to identify the pathway that is relevant to them in the JusticeBot, even though some of these situations are already covered in the JusticeBot. Thus, by annotating these user-submitted descriptions, we can evaluate whether the system is able to surface a pathway that is relevant to them. The experimental setup is described below in Section 3.2.
tions, we embed the stored examples using a sentence encoder into a vector format, and then use an approximate nearest neighborhood model to retrieve sentences that are semantically similar to the user description.
To convert each sentence into an embedding, we use a multilingual universal sentence encoder. This model is a multi-task, dual-encoded [45] convolutional neural network, that has been pre-trained to embed texts from 16 languages into the same semantic embedding space [39]. We use a pre-trained version of this model available on tensorflow-hub. 3 The model takes a sentence as input, and produces a vector of 512 dimensions, capturing the semantic content of the text.
database, we use the Annoy similarity search library released by Spotify4. We opted for this solution for its ease of use and minimal system requirements. The annoy library is a very quick and light implementation of an Approximate Nearest Neighbors algorithm proposed in [46]. The library enables us to build an index for the sentence embeddings created in the previous step. When supplied with a vector, the index can surface the N most similar vectors, in a fraction of a second.
Once they pause writing, the text entered by the user is sent to the server. This means that the user can get relevant explanations even before they have completed the typing of their description, which saves time since they may already see the relevant suggestion after starting to write, making it unnecessary to complete their writing.
On the server side, once the factual description is received, the description is vectorized with the embedding model described above in 3.4. Then, the search index is used to retrieve the previously embedded examples, as described in section 3.5. This results in a list of the example descriptions that are the most similar to the user factual description. Since these example descriptions are linked to a relevant legal issue (see 3.3), we can retrieve the legal issue that was seen as relevant for an example description. The top three legal issues that have an example description linked to them that is the most similar to the user factual description are shown to the user.
The user is shown the factual explanation and suggested action (see Section 3.1). They can verify that the factual explanation corresponds to their situation, and click on the suggestion to be taken to the relevant page in the JusticeBot, or to an external source that provides useful information to them.
An advantage of the technique used in this approach is that it is very fast. The entire analysis performed on the server (i.e. embedding the user factual description, retrieving the most similar example descriptions and selecting the relevant suggestions) can be done in a few milliseconds. This means that the user can obtain suggestions very quickly.
3
Users interacting with the system provide valuable additional training data for the model, that can be verified by expert annotators. If a user writes the description of their facts into the text box that can be seen in Figure 2, they receive suggestions from the system. While some of these suggestions will be relevant, some will not. How the user acts in response to the suggestion can be a strong indicator of whether the suggestions are useful or not.
When given suggestions, users can take three actions: 4.1. Dataset • They can click one of the suggestions. This indicates that their factual description is likely relevant to the link that they clicked. Thus, the description can be saved and potentially be included in the system as an example description. • They may not click any of the suggestions, and instead click one of the links in list of pathways shown below. This means that the suggestions surfaced by the system were not relevant in this case, and that the user opted to use the standard list for selecting their pathway. In this case, their factual description can still be used, and added to the suggestion of the pathway that the user selected. • If the user enters a factual description, and then clicks on the “Other” heading, this may be an indication that none of the options are satisfactory to them. Thus, the description of the user can be added to the “missing questions” database, as described in 1.
To evaluate our research questions, we focused on the perspective of an individual who has indicated that they are a tenant, as shown in Figure 1. First, for each of the available pathways, we created seed example descriptions, by imagining how users might express themselves regarding situations that would benefit from the pathway. Further, we created some suggestions that take the user deeper into the pathway—for example, the issue of heating is linked to a pathway that allows the user to explore a rent reduction or lease termination. These example descriptions are referred to as “seed” in Table 1. We created them in french, but note that due to the Thus, the system can collect data and improve over time, multilingual nature of the embedding model used, even as users interact with it. Of course, the data would need seed examples in other languages should be usable by to be anonymized, and the user be made aware that their the system. input can be used in this way. Next, we analyzed the factual descriptions submitted
We have now seen the approach we developed to link by users as not covered by the JusticeBot (see Section 1). users’ factual descriptions to legal issues. Next, we will For the issues that are covered by the JusticeBot, we noted describe how we evaluated this approach. the corresponding pathway and added the examples to the dataset. In total, we annotated 3,250 such submitted 4. Experimental design feedback examples. These range from a single word to multiple sentences. This part of the dataset is referred In order to evaluate the methodology described in Section to as “user” in Table 1. As can be seen in Table 1, we 3.2, we analyze three research questions: identified a substantial amount of situations where the 1. RQ1 - Can the proposed methodology achieve ade- JusticeBot contains relevant information, but the user quate performance in pointing individuals toward was not able to find the relevant pathway. the correct legal issues using the seed examples In analyzing the submitted feedback, we also encounonly? tered many situations that were not yet covered by the 2. RQ2 - Can the performance of the methodology in Justicebot. As discussed in [12], this serves as an excelguiding individuals toward the correct pathway lent basis for prioritizing the addition of new pathways. be further increased by augmenting the database Further, links to external content can be introduced as with real-life user factual descriptions? a stop-gap measure that can already help, even if the 3. RQ3 - Can the use of a language model and seed JusticeBot pathway is not yet available. While this will examples overcome the cold-start problem to need to be further explored, we already added three such rapidly achieve usable performance? pathways that seem to frequently re-occur, namely questions regarding animals, repossession and nuisances (see Table 1). For each of these, we created seed examples and assigned user examples. 4.2. RQ1 - Our methodology could achieve adequate performance in pointing individuals toward the correct legal issues, using only seed examples.
validation setting. As such, the “training data” consists of the seed data and the user-submitted data minus one sample, while the “test data” consists of the held-out sample.
We report the scores of whether the correct suggestion is displayed first (P@1) and whether the correct suggested pathway is part of the top 3 results shown to the user (P@3).
Our first research question explores if the proposed methodology is able to leverage the power of the language model used to achieve strong performance when retrieving legal issues, even if only the seeded examples The third research question analyzes whether the lanare used. This would be a strong indication that the NLP guage models used in the methodology can bring the benfeature could be useful even if no user data has been efit of quickly adapting to the training data. Traditional collected, and thus that new user needs can be quickly machine learning often runs into the cold start problem, addressed as they arise. Of course, for this to work, the where it takes a while to be able to learn enough to genermethodology needs to be able to draw links between the alize beyond the training data. RQ3 investigates whether comparatively clean and structured seed examples and our approach can help overcome this issue. This could the real-world descriptions written by laypeople, that be the case due to our use of language models, which are likely to be much more varied in terms of factual have been trained on large corpora to absorb language situation, content and tone. patterns. Further, we use seed examples, that are syn
To investigate this research question, we trained a thetically created training examples aimed to teach the model on only the seed example descriptions prepared models some patterns before real data can be collected, by our team. For each user-submitted factual description, which could also help in overcoming the cold-start probwe test whether the correct suggestion is surfaced first lem. (P@1) or in the top 3 suggestions shown to the user (P@3), To investigate this issue, we perform an experiment i.e., whether the correct pathway is suggested to the user where we continually add training data and see how in the first place, or visible at all in the interface shown in the performance of the model develops (compare [47]), Figure 2. As such, here the “training data” used consists when tested against test data (100 random samples of the of the seed example data, while the “test data” consists user-submitted data that we withheld). We compare our of the user-submitted example descriptions (see Table 1). language model-based approach to a more traditional strong baseline in the form of a support vector machine (SVM) trained on TF-IDF representations of the example descriptions. For both models, we run two variants of the test: one where the model is first trained on the seed examples and then on user-submitted examples, and one where it is only trained on the user-submitted examples (user_only).
methodology to guide individuals toward the correct pathway can be further increased by adding real-life user factual descriptions.
tional factual examples, provided by the users, to “train” 5. Results the system will increase the amount of correct suggestions. This would indicate that adding additional, real- 5.1. RQ1 - Our methodology could world data to the system increases its performance, i.e., achieve adequate performance in that it improves over time. pointing individuals toward the
To investigate this research question, we trained a correct legal issues, using only seed model based on the seed data and the user-submitted factual descriptions. Then, for each user-submitted fac- examples. tual description, we use the methodology to retrieve the Table 2 shows the performance of the model when trained suggested pathway. Of course, we excluded the user- on the seed examples prepared by our team only. Overall, submitted pathway that is currently used in retrieval 54.8% of the users would be provided with a suggestion when evaluating the result, i.e. in a leave-one-out cross- that is relevant to them by the system. The performance examples only. Overall, 74.5% of user-submitted queries would have received the relevant suggestion at the top spot, while 93.5% of users with one of the captured situations would have received a relevant suggestion in the three suggestions that they were shown. Only 6.5% of users would not be shown a relevant suggestion. The results vary between the diferent issues—for lease transfer, about 30% of the users would not see the relevant suggestion, while for sublease, 100% of users would see the relevant suggestion in the 3 entries surfaced to them.
Figure 3 shows the performance impact of using our language model-based approach versus a support vector machine, with or without the seed examples. As we can see, our approach reaches reasonable performance much quicker than the SVM. For the SVM, the use of seed examples increases the performance and the speed at which the model can adapt to the real-world data. 6. Discussion varies widely between the diferent legal issues. For the animals category, inquiring about questions regarding animals in an apartment, almost 80% of the users would have received a relevant suggestion. For a bedbug infestation, on the other hand, only a third of the users would have received the relevant pathways. We will discuss this divergence in Section 6.
methodology to guide individuals toward the correct pathway can be further increased by adding real-life user factual descriptions.
legal issues from layperson factual descriptions, and presented the results of our evaluation. Let us discuss these Table 3 shows the performance when the system is results, and some aspects of the proposed methodology “trained” on the seed example description as well as user- in increasing access to justice. submitted example descriptions, and evaluated on one of the user-submitted examples at a time. As we can see, the performance is much higher than when using the seed 6.1. RQ1 - Our methodology can achieve adequate performance in pointing individuals toward the correct legal issues, using only seed examples.
queries definitely enhances the performance of the model, compared to when using seed explanations only. This shows that such a system built using real-world data could work well in practice, and contribute to increasing access to justice.
As we can see, the proposed approach is able to surface 6.3. RQ3 - Using a language model and the correct legal issue in the top 3 suggestions for 54.8% seed examples can overcome the of the user descriptions.
This may seem like a relatively low number. However, cold-start problem to rapidly achieve it is important to consider the dificulty of the performed usable performance task. First, we only use the very few seed examples to In our third research question, we investigate whether “train” the model. For each suggestion, there are only language models and seed questions can overcome the 3–9 such seed examples. Further, the tasks consist of an- cold-start problem. This seems to be the case, as the alyzing layperson language and linking it to structured language models is much quicker to learn a pattern from legal issues. Layperson texts are notoriously dificult to the provided data than the SVM, both in the configuration deal with. For example, in [6], the authors found that with and without seed data. Likewise, when the seed data submissions by unguided pro-se litigants could not be is used, the SVM is quicker to adapt to the real-world used to predict the outcome of cases, likely due to the user-submitted data. Thus, it seems like both of our gap between common parlance and legal language. Even approaches paid of and show promise in overcoming though we attempted to create seed examples that are the issue of a cold start. However, it also seems that similar to how users are likely to describe their issues, potentially another approach (such as SVM or BERTit is possible that our one-sentence examples are very based classifiers) should be used once enough data has diferent from the way real-world users describe their been collected, to guarantee the best performance. Future issues. For example, the average length of the seeded ex- work will investigate how other models will perform at amples ( 51 characters) is less than half the length of the diferent stages, and when the switch should be made. user-submitted examples ( 109 characters). Further, the factual situation mentioned in the user-submitted examples are likely to be much more diverse than the limited 6.4. Augmented Intelligence number of situations captured in the seed examples. The question remains whether the performance shown
From this viewpoint, the model being able to identify by these models is “adequate” for deployment in a realthe correct legal issue in over half of the cases is a rea- world model. A factor that is important to consider is sonable result. It shows the power of the language model that the feature as described here is conceptualized as an used to absorb the semantic meaning of the sentences. “augmented intelligence” approach, that aims to support Of course, there is room for improvement, which will be the user instead of replacing them. It does not automatexplored in future work. ically take the user to a pathway, but instead surfaces three suggestions that may be relevant to the situation 6.2. RQ2 - The performance of the described by the user. Each suggestion has a “factual exmethodology to guide individuals planation”, that allows the user to verify that the model toward the correct pathway can be correctly analyzed their factual situation. If the factual explanation of the suggestion matches the situation of further increased by adding real-life the user, the legal issue is guaranteed to be relevant, as user factual descriptions. it has been encoded by legal experts. Thus, the user is given a way to meaningfully verify whether the system worked, without having to understand the legal particularities of their situation — they merely have to read the “factual explanation” and verify that the system correctly understood their situation.
Thus, the “cost” of a failure of the system is relatively low. If the user enters a description of their situation, and none of the suggestions seem relevant, they can simply ignore the feature and continue using the JusticeBot.
However, if the system surfaces a relevant suggestion, a user can be directed to the appropriate pathway, that
previous user-submitted data is used to “train” the model.
The overall performance is strong, with 93.5% of the usersubmitted factual descriptions surfacing the correct legal issue (P@3). The performance varies somewhat between the diferent classes (between 70% and 100% at P@3), which may be due to the “semantic homogeneity” of the classes - perhaps, certain legal issues have much more divergent situations and descriptions that could give rise to them (compare [47]).
We observe that adding data from the user-submitted they otherwise may have missed. Even if a fraction of information, by merely supplying context to the user, and the users benefits from using the suggestions described letting them make the relevant decisions. The feature dein this paper, access to justice could be improved. scribed in this paper, however, goes a bit further than this,
While, of course, additional empirical evaluations need by providing the user possible legal pathways that they to be undertaken to make sure that the use of the system can explore, based purely on a factual description. An is safe, this framing may make the achieved performance argument could be made that this could be seen as giving “adequate” for an initial deployment. Further, once the legal advice. However, looking at the information prosystem is live, the data collected can be a powerful way vided in Figure 2, it is important to note that the system to enhance the system, as described in Section 3.7. is focused on augmenting the intelligence of the user—it is the user that makes the decisions. The user decides 6.5. Potential benefits of the approach whether the surfaced suggestion makes sense and thus which pathway they want to explore. Further, clicking the suggestion leads back to the JusticeBot, which only provides legal information.
As we have seen, the proposed system appears to perform well in identifying the legal issues from real-world, usersubmitted factual descriptions. Integrating such system into a decision support tool, such as the JusticeBot, could play an important role in overcoming the gap between regular language and legal language, and give laypeople a better chance to understand how the laws apply to them or the legal remedies available to them. This could be an important step towards increasing access to justice.
At the same time, it is important to keep in mind that the evaluation presented in this paper is limited to examples where the situation described by the user is already part of the JusticeBot. If the situation the user describes is not covered by a suggestion, they would be provided with a list of irrelevant pathways. However, as described above, they would be able to realize this, by seeing that the factual explanation does not apply to them. Even so, in expanding the coverage of legal decision support tools and the JusticeBot, it remains important to add new pathways for frequently recurring situations, or to add references to external content, to ensure that as many people as possible can be helped. The data collected through the use of the NLP feature could be an important tool in determining which pathways to add (see Section 3.7). We also plan to experiment with setting a threshold for sentence similarity that needs to be exceeded for suggestions to be returned at all, which should minimize providing obviously irrelevant suggestions to the users.
We anticipate that the feature described in this paper would also work well for other approaches, beyond the JusticeBot. Connecting layperson language to legal issues is important in many legal tasks. Being able to automate this process, even partially, could lead to many interesting projects e.g. in automated document drafting.
One also has to be cognizant of other risks of using the proposed approach, including the prohibition against the unauthorized practice of law in many jurisdictions. The distinction between providing legal information and legal advice may not always be clear [48]. The JusticeBot itself is specifically designed in a way to only provide legal
the dataset needs to be expanded, beyond the currently limited number of issues. Adding new suggestions concerning other frequently occurring situations could expand the usefulness of the tool and provide more context for evaluation. Large language models (LLMs), such as GPT-4 [49], have been used to generate legal example situations [21] and perform legal annotation tasks [50, 51], and may thus be an important way to make this task more eficient. The system should also be evaluated in the context of JusticeBot tools in other legal domains. Second, other embedding approaches or nearest neighbor search methods could be tried. Likewise, other machine learning models (including LLMs such as GPT-4) could be used. Third, applying the method described here to other tools would be an interesting way to explore how generalizeable the approach is. Fourth, pilot studies with end-users could help us understand the real-world utility of the system and identify further areas of improvement.
scriptions to legal issues. We described and discussed the approach. The initial evaluations on real-world user data are promising, and could represent an important addition to the JusticeBot methodology and access to justice.
John, N. Constant, M. Guajardo-Cespedes, S. Yuan, [40] G. Lai, B. Oguz, Y. Yang, V. Stoyanov, Bridging the
C. Tar, et al., Universal sentence encoder, arXiv domain gap in cross-lingual document
classificapreprint arXiv:1803.11175 (2018). tion, arXiv preprint arXiv:1909.07009 (
preprint arXiv:1607.01759 (2016). [49] OpenAI, Gpt-4 technical report, 2023. [36] A. Conneau, D. Kiela, H. Schwenk, L. Barrault, a r X i v : 2 3 0 3 . 0 8 7 7 4 .
A. Bordes, Supervised learning of universal sen- [50] J. Savelka, K. Ashley, M. Gray, H. Westermann, tence representations from natural language infer- H. Xu, Can gpt-4 support analysis of textual data ence data, arXiv preprint arXiv:1705.02364 (2017). in tasks requiring highly specialized domain exper[37] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: tise?, in: ASAIL’23: 6th Workshop on Automated Pre-training of deep bidirectional transformers for Semantic Analysis of Information in Legal Text, language understanding, in: NAACL 2019, Volume 2023.
1 (Long and Short Papers), 2019, pp. 4171–4186. [51] J. Savelka, Unlocking practical applications in le[38] A. Conneau, K. Khandelwal, N. Goyal, V. Chaud- gal domain: Evaluation of gpt for zero-shot sehary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, mantic annotation of legal texts, arXiv preprint L. Zettlemoyer, V. Stoyanov, Unsupervised cross- arXiv:2305.04417 (2023). lingual representation learning at scale, in: ACL 2020, 2020, pp. 8440–8451. [39] Y. Yang, D. Cer, A. Ahmad, M. Guo, J. Law,
N. Constant, G. H. Abrego, S. Yuan, C. Tar, Y.-H.
Sung, B. Strope, R. Kurzweil, Multilingual universal sentence encoder for semantic retrieval, 2019. a r X i v : 1 9 0 7 . 0 4 3 0 7 .