SCaLe-QA is a foundational system developed for Sri Lankan Legal Question Answering (LQA) by leveraging domain-specific embeddings derived from Supreme Court cases. The system is tailored to capture the unique linguistic and structural characteristics of Sri Lankan law through fine-tuned embeddings. While Case-Based Reasoning (CBR) will be integrated into the question-answering framework, it is primarily set for future development and evaluation. Currently, SCaLe-QA employs semantic chunking, tokenization, and BM25-based ranking to generate context-driven triplets from unlabeled corpora. In addition, an angle-optimised contrastive learning framework is applied to enhance retrieval accuracy. Preliminary results indicate promise, establishing SCaLe-QA as a significant step toward robust AI applications in the Sri Lankan legal domain.
The increasing complexity of legal texts, particularly within the Sri Lankan judicial system, poses
significant challenges for the development of efective Legal Question Answering (LQA) systems. Legal
documents are characterised by specialised vocabulary, intricate syntactic structures, and
contextdependent semantics, making the task of automated question answering both demanding and essential.
The ability to accurately and eficiently answer legal questions is critical, as it enhances access to
legal information, supports legal research, and facilitates informed decision-making processes for legal
professionals, researchers, and the general public [
Recent advancements in Natural Language Processing (NLP) and Machine Learning (ML) have spurred
the development of sophisticated LQA systems that leverage deep learning techniques to process and
understand legal texts efectively. These advancements have been well-documented, highlighting the
importance of domain-specific datasets and models tailored to the unique characteristics of legal texts
[
Retrieval Augmented Generation (RAG) has emerged as a powerful approach in enhancing the performance and reliability of LQA systems. RAG combines the strengths of retrieval-based methods with generative models, allowing for more accurate and contextually relevant responses to legal queries. This approach involves retrieving relevant documents or passages from a large corpus of legal texts and then using this retrieved information to augment the generation process [4]. Furthermore, the integration of case-based reasoning (CBR) systems with specialised embeddings as investigated in [5] has been shown to improve the performance of such systems compared to typical information retrieval techniques [6] in the legal context.
SICSA REALLM Workshop 2024
CEUR
ceur-ws.org
In this contribution, we explore the impact of tuning domain-specific embeddings for legal contexts, focusing on how these embeddings can be utilised to transform triplets from unprocessed legal documents into structured representations. Our aim is to improve retrieval accuracy within Retrieval Augmented Generation (RAG) systems, which is crucial for the efectiveness of Legal Question Answering (LQA) systems. By enhancing these embeddings, we aim to significantly boost the performance and reliability of LQA systems tailored to the legal domain, particularly in the Sri Lankan legal space.
The dataset used for this study consists of the oficially reported Supreme Court judgments from Sri Lanka, spanning from 2009 to 2024. In total, 1541 documents were scraped from the oficial Supreme Court website1. The documents covered a wide area of the Sri Lankan legal context such as: • Appeals: Includes general appeals such as standard appeals, civil appeals, and specific appeals related to legal provisions or leave to appeal applications. • Applications: Divided into constitutional applications, fundamental rights applications, and other various legal procedure applications. • Civil Cases: Encompasses general civil matters, including commercial and procedural cases, as well as specific civil cases like divorce, testamentary, and land disputes. • Criminal Cases: All cases related to criminal law. • Constitutional Matters: Includes cases dealing with constitutional law, references, and specific declarations under the Constitution. • Commercial High Court Cases: Covers commercial disputes handled by the Commercial High
Court. • Other: A variety of other case types, including contempt of court, election-related matters, and writs of certiorari and prohibition.
Most of these documents were directly text-parsable, while others required additional processing. The non-parsable documents were fed into an OCR model using Adobe 2, and the resulting text was manually corrected to ensure accurate extraction. This cleaned and corrected dataset served as the primary data source for the subsequent stages of this research. 1https://www.supremecourt.lk/ 2https://www.adobe.com/in/acrobat/online/ocr-pdf.html
In this work, we followed the document chunking strategies from the Open Australian Legal QuestionAnswering (ALQA) dataset3, which is based on legal question answering for Australian law. The document chunking strategy served as a preprocessing step, assisting the sentence tokenizer in breaking segments into sentences. More importantly, it established the foundation for creating the testing framework necessary for the embedding fine-tuning process, which will be discussed later. We employed the semantic chunking model provided by the SemChunk library4, which was particularly useful for handling legal documents. We maintained consistency with the Australian Legal QA dataset by setting the chunk size to 384 tokens, as tokenised according to the tiktoken tokenizer for GPT-45. Upon manual inspection, the chunk size was deemed appropriate, and it did not negatively impact sentence integrity.
To gain insights into the dataset, after preprocessing, we performed some sentence-level visualisation as shown in Figure 2. The visualisation highlights that the dataset includes very long documents, with some exceeding 1500 sentences. These visualisations emphasise the significant variability in both the size of documents and the length of sentences, which presents unique challenges for processing and analysing legal texts.
Each sentence after sentence segmentation is treated as individual units for subsequent processing to create triplets. The BM25 algorithm, a robust ranking function used in information retrieval [7], is applied to rank these sentences based on their relevance to each other within each case document. This method leverages the lexical knowledge embedded in the text as weak supervision, which has been shown to be a strong baseline for fine-tuning text embeddings, as discussed in [ 8]. The BM25 algorithm was implemented using the rank_bm25 library.6, and the algorithmic breakdown for the preprocessing will be shown below.
Given a sentence in a document , the BM25 score for another sentence ′ in the same document is calculated using the following adaptation of the BM25 formula: score( ′, ) = ∑ IDF( ) ⋅ =1
( , ′) ⋅ (1 + 1) ( , ′) + 1 ⋅ (1 − + ⋅ av| g′d|l ) (1) 3https://huggingface.co/datasets/umarbutler/open-australian-legal-qa the length of the sentence and avgdl is the average length of sentences in the document. 1 and are hyperparameters (typically 1 = 1.5 and = 0.75 ), and IDF( )is the Inverse Document Frequency of term .
In this approach, for each sentence in a document, we rank all other sentences ′ in the same document using the BM25 algorithm. The most similar sentence (i.e., the one with the highest BM25 score) is selected as the positive or like sample . To select a negative sample or unlike sample , we randomly choose one out of the top five least similar sentences (i.e., those with the lowest BM25 scores). This strategy helps avoid the issue of a very dissimilar sentence being repeatedly included in the triplets, which could make the triplets less informative for training. The number of least unlike sentences considered is obtained through empirical experimentation.
The sentences identified through this process are then denoted as follows: • : The anchor sentence, which is the sentence we are evaluating within the document. • : The positive sample, the sentence most like the anchor.
• : The negative sample, a sentence sampled from a pool of N unlike (least similar to) the anchor.
These notations will be used in the subsequent embedding fine-tuning process.
A contrastive compound loss function was designed to optimise the distances within the embedding space between triplets ( , , ), where is the anchor sample, is the positive sample (similar or like the anchor), and is the negative sample (dissimilar or unlike the anchor). The compound loss function is inspired by the methodologies in [9], where angle optimisation is combined with the cosine objective from [10]. This approach diferentiates itself from existing contrastive learning methods discussed in [11, 12]. Specifically, the loss function combines three key objectives: = 1 ( , ) + 2 (− ∑ ∑ log ( )) + 3 ( ′ , ′ )
(2) exp ( ) ∑ exp ( ) similarity than negative pairs.
• The first term ( , ), weighted by 1, uses the standard cosine similarities between the anchor and positive (or like) instance, = (
, ), and the anchor and negative (or unlike) instance, = (
, ). Here the general contrastive loss function is defined as: ( , ) = log (1 + ∑exp ( − )). This encourages the model to ensure that positive pairs have higher • The second term, weighted by 2, applies in-batch negative sampling, comparing the anchorpositive pairs within a batch and treating the remaining pairs as negatives. Again using cosine similarities to arrive at and respectively.
averaging the cosine similarity over the two halves of the embeddings. • The third term weighted by 3, is similar to the first but uses a refined similarity metric, the embeddings of , and are split in half. The similarities ′ and ′ are calculated by ′, where Here : is a temperature scaling parameter that controls the sensitivity of the model to diferences in similarity scores. Lower values of increase the sharpness, while higher values soften the distribution. Parameters and represent the batch size and number of batches, respectively.
The loss function was applied to fine-tune AnglE-BERT, a model specifically designed for angleoptimised contrastive learning, as introduced by [9]. AnglE-BERT was initially trained with the angle optimisation mechanism that adjusts the angles between embeddings in the latent space, which is particularly efective for distinguishing between similar and dissimilar pairs of sentences. Each finetuning run for AnglE-BERT was carried out using the triplets formed from the BM25-ranked sentences, executed on an NVIDIA RTX A100. The training phases were conducted with a batch size of 32, over 10 epochs, spanning around 14 GPU hours, during which the model was exposed to just over 230,000 triplets. The extensive training was aimed at refining the model’s ability to learn the nuances of the legal texts seen in the triplets.
As introduced in AnglE-BERT [9], two distinct flavours of embeddings were fine-tuned using contrastive learning, each optimised for diferent retrieval purposes. In this work, these embeddings will be categorised as Intra-Embeddings and Inter-Embeddings, which are designed to serve specific retrieval and matching tasks.
• Intra-Embeddings (f(Q)): These embeddings are optimised for attribute matching within the same type of content, such as comparing questions with questions. This type of embedding is particularly useful for semantic textual similarity tasks, where the focus is on finding sentences with closely related meanings, even if they are phrased diferently. • Inter-Embeddings (g(Q)): These embeddings are designed for broader information retrieval scenarios, where the goal is to match content across diferent types of attributes, such as comparing a query with relevant passages, entities, or supporting texts. This allows for more flexible retrieval tasks, where the query might need to be matched with various types of contextual information.
The fine-tuning process outline before for embeddings was conducted separately on both the intra and inter embeddings, ensuring that each representation was optimised for its respective task. This dual fine-tuning approach allows the model to perform well across both precise attribute matching and broader retrieval tasks. Conceptually, this approach is akin to a form of query rewriting[13], where each type of embedding acts as a diferent representation of the input query, tailored to optimise retrieval for specific purposes.
Table 1 provides an example illustrating the diference between intra-embedding and inter-embedding for a sentence used in the training process.
Prior to evaluating the performance of the embedding models, there are two key stages: Casebase Creation and Test Set Creation. These stages are crucial for building the necessary datasets for evaluating the retrieval models in the subsequent retrieval evaluation.
The first stage in the evaluation process involved constructing the casebase, using the scraped Supreme Court documents. As illustrated in Figure 3, this process involved multiple steps of attribute extraction from legal documents. The documents were segmented into manageable chunks of 384 tokens, and the following key attributes were extracted using the GPT-4o mini7 model: 7https://platform.openai.com/docs/models/gpt-4o-mini • Court Details: Including court name, case number, case year, and case type. • Parties Involved: Identifying plaintifs, defendants, and their respective roles. • Questions of Law: Legal questions presented in the case, particularly those considered by the
Supreme Court. • Case Summary: A brief summary of the case, including judgment details, legal issues, and key ifndings. • Laws and Acts Referenced: Listing specific laws or legal acts cited during the judgment. • Judgment Details: Including the decision outcome, key findings, and legal conclusions reached by the court.
Once extracted, these attributes were compiled into structured JSON records, providing a metadatarich view of each case. This enabled eficient question-answer generation by providing the necessary context of the full legal case in a condensed structure, for each document chunk, rather than requiring the system to process the entire case document.
The second stage involved creating a robust test set from the casebase. This process, depicted in Figure 4, starts by filtering the entire casebase to group cases that share overlapping legal references, such as common laws or acts. Metadata filtering was applied in such a way that for each case in the casebase, other cases with overlapping laws were identified. These similar cases were ranked based on how closely related they were, but ensuring that they originated from diferent legal documents, thus enhancing diversity in the source material.
The cases selected through this filtering process were further refined by applying a cosine similarity ranking mechanism, using the Ada-002 embedding model [14] to identify closely related case pairs using the query of each case. These pairs were then used to generate a new hybrid question and answer through a prompt designed for the GPT-4o 8 LLM. The generated question was complex and required the context of both related case snippets to be answered correctly.
A human-in-the-loop system was employed to review the generated question-answer pairs. The evaluators (authors of this paper) assessed the quality of each pair, filtering out those that lacked relevance or quality. This rigorous review ensured the creation of a high-quality test set for further retrieval evaluation.
This test set contained 1000 high-quality question-answer pairs to evaluate the embedding retrieval. 8https://platform.openai.com/docs/models/gpt-4o
The embedding model performance was assessed using multiple Retrieval@K evaluations, which helps in understanding how well the model ranks and retrieves relevant information based on the hybrid test cases. The retrieval evaluation included analysing both the F1-score@K and Recall@K, which provide insights into the balance between precision and recall during the retrieval process.
To conduct this evaluation, we used k-Nearest Neighbors (k-NN) based retrieval, exploring a range of values between 1 and 37. These prime values allowed us to investigate the optimal retrieval size for the legal documents used in the study.
We evaluated our fine-tuned AnglE-BERT model for both intra and inter-embeddings, comparing it against the standard BERT[15] and AnglE-BERT models[9]. Figure 5 shows the heat maps comparing Recall@K and F1-score@K for these models and their respective weight configurations.
The results indicate that fine-tuning on AnglE-BERT improves both the Recall@K and F1-score@K across diferent retrieval levels. Specifically, the fine-tuned AnglE-BERT model with diferent weight configurations has shown a robust retrieval performance whereas AnglE-BERT has performed well in only query-to-query matching. These findings suggest that the fine-tuned model’s ability to retrieve relevant legal cases was robust and well-adapted to the unique characteristics of the legal domain.
The embedding distribution, as illustrated in Figure 6, was obtained by calculating the cosine similarity between the query (i.e., the question for each case) and its corresponding snippet or context.
For the standard BERT and AnglE-BERT models, the similarity distribution is skewed to the left. This left-skewed distribution indicates that these models classify more query-snippet pairs as having a relatively high similarity score. This behaviour suggests that BERT and AnglE-BERT may not be capturing the nuanced relationships between legal queries and snippets efectively, potentially leading to a higher number of false positives in retrieval tasks. In contrast, the fine-tuned AnglE-BERT model exhibits a more normal-like distribution. This shift suggests that the fine-tuning process has improved the model’s ability to diferentiate between relevant and irrelevant cases, balancing the similarity scores across query-snippet pairs. The more centred distribution may be an indicator that the fine-tuned model is better adapted to the legal domain, likely learning the domain-specific vocabulary and complex semantic relationships within legal texts. As a result, it performs more robustly in distinguishing between cases with subtle variations in meaning, leading to improved retrieval performance.
In this work, we developed SCaLe-QA, a foundational system tailored to the specific requirements of Sri Lankan Legal Question Answering (LQA) tasks by using domain-specific embeddings derived from Supreme Court cases. Our work primarily focused on enhancing the retrieval accuracy of a RAG system using CBR by fine-tuning embeddings, using BM25 ranking for triplet generation and contrastive learning methods. An interesting finding was the creation of dual representations for the query depending on the attributes being compared for retrieval. Finetuning in this manner resulted in superior F1 scores. Future work will involve integrating Case-Based Reasoning (CBR) to build more comprehensive question-answering models, as well as expanding the scope of SCaLe-QA to attribute-focused embedding models.