Introduction

In the last years the interest for Generative Artificial Intelligence (Generative AI) applications has grown importance among the research and industry community, thanks to the introduction of Foundation Models in different AI domains, such as text generation (GPT-series [1,2,3], LLaMA-series [4,5,6], MEGATRON [7]), image generation (DALL• E [8,9,10], Stable Diffusion [11]), and video generation (Sora [12]). The progress in Deep Learning modelling has been fostered by important advancements in the neural network (NN) research, such as the introduction of the Transformer architecture in Nat-ural Language Processing (NLP) [13], by improvements in hardware acceleration for linear algebra, expanding model's size up to several billions of parameters, by the introduction of quantization techniques allowing training of large NNs even on consumer GPUs [14,15], and by the release of large high-quality and open-source datasets [16].

Large Language Models (LLMs) for text generation have achieved remarkable performance and great interest even outside the research and industry professional community, in particular after the release of ChatGPT to the public [17]. Despite their success, the use of LLMs on domain-specific Question-Answer (QA) tasks still face several challenges, that hinder their spread beyond the research community, especially when applied to tasks for which explainability and high quality responses are of paramount importance [18]. Some of the challenges that LLMs are still facing are the following:

• difficulty of maintaining up-to-date knowledge • costs of training and inference of large models, costs and difficulty to collect large amount of highquality domain-specific data • hallucinated answers, i.e. answers that provided false information without warning

• out-of-date or generic answers, even when the user expects a specific, current response Retrieval-Augmented Generation or RAG has recently emerged as a paradigm to address such challenges [19]. In particular, RAG combines a language model with an information retrieval system to dynamically fetch relevant external information to enhance the model's responses, by encoding the user's question into a dense representation, and retrieving passages relevant to the question from an indexed data source, adding this information to the LLM prompt. Different studies have shown that RAG enhances the quality of the generation process, leading to higher accuracy, better robustness, reduced hallucinations, higher interpretability, and even the possibility to perform open-domain QA just by updating the knowledge-base [20,21]. RAG also offers a balanced approach in terms of customization and resource requirements, being more flexible and cost-effective than full fine-tuning, although still requiring labeled data and a supervised training phase.

In this work we present SAVIA, a project developed by CINECA and Assemblea Legislativa of Emilia-Romagna. This project, started in Autumn 2023 and expected to end in march 2025, has the goal of creating a model capable of answering questions on the Region's law and their respective implementing acts, as well as on related "exante" and "ex-post" reports of the laws' impact. In Section 2 we present the data used for this project and the workflow that has been adopted. In Section 3, we describe the procedure and the details of the experiments and tests conducted, and in Section 4 we show the obtained results. Our conclusion are then presented in Section 5.

Methodology

To obtain a model capable of understanding Italian language in the law domain and responding to questions related to laws enacted in the Emilia-Romagna region, we followed a multi-step approach. We started from an open-source LLM and adapted it to the legal language through unsupervised domain adaptation (Section 2.1). The resulting domain-adapted model was then fine-tuned for question-answering (Q&A) on an instruction-based dataset prepared by domain experts for this purpose (Section 2.2). Finally, we implemented a domain-adapted retrieval model (Section 2.3) to enrich the answers with relevant information from the law corpus.

The full workflow was reproduced starting from different open-source LLMs:

• LLaMAntino-2-7b-hf-ITA: a 7B model, based on LLama-2, specifically fine-tuned for the Italian language [22].

• Mistral-7B-v0.1: a 7B model, that implements grouped-query and sliding window attention, Rotary Position Embedding, that can handle context of arbitrary size [23]. • Mixtral-8x7B-Instruct-v0.1: a 46.7B mixture of experts model, trained on instructions in English, French, Italian, German and Spanish, with maximum context length of 32k [24].

Domain experts qualitatively evaluated the performances of the final models obtained from the different pre-trained LLMs.

Unsupervised Domain-Adaptation

The first step in the procedure was the domain-adaptation of the model on legal text. We collected the PDFs of the regional laws of Emilia-Romagna, as well as the relative implementing acts at the regional level (e.g. "atto del dirigente" and "atto di giunta") and the available reports on the expected and measured impact of a given law (e.g. "clausola valutativa", "ex-ante and "ex-post" reports). We split the legal documents in chunks, and we implemented a cleaning pipeline to remove typos, bad characters, and irrelevant parts of the documents such as headers and footers. We also added mii-llm/gazzetta-ufficiale [25] in the training dataset, given the affinity of this dataset to our application, both in language, semantics and type of documents. We did not perform domain-adaptive tokenization [26], using instead the pre-trained models native tokenizers to tokenize the legal corpus.

Not all the three model under investigation underwent domain adaptation. LLaMAntino-2-7b-hf-ITA and Mistral-7B-v0.1 were adapted, while Mixtral-8x7B-Instruct-v0.1, after tests regarding its native capabilities of producing adequate Italian legal text, has not been domain adapted.

Model Alignment on Instruction-Based dataset

With the support of domain experts, we generated an Q&A dataset mimicking different levels of domain language proficiency, ranging from questions that could by written by non-expert users, to the ones that may be asked by experts in the legal domain. We developed a semi-automatic procedure to further enrich this Q&A dataset, using legal documents metadata. The following is an example included in the instruction-based dataset:

• Q: "Da quando è stata istituita la regione, quali normative sono state adottate per incentivare la partecipazione?" • A: "La prima legge regionale riguardante la partecipazione ad essere stata approvata è la legge numero 3 del 2010. In seguito, la legge numero 3 del 2010 è stata abolita e sostituita con la legge regionale numero 15 del 2018. ", To fine-tune the domain-adapted LLMs, we used the instruction-based dataset prepared by the domain experts. For the loss function computation, we removed the portion of the text containing the prompt, as in many cases the prompt added by the RAG module can account for up to 50% of the total text length. This approach helped to optimize the training process more effectively.

Domain-Adapted Retrieval Model

To enrich the user's question with relevant information from the legal documents database, we develop a retrieval module based on Semantic Similarity search technique. We used a Sentence-BERT model [27] to populate a vector store with embedding generated from the legal documents text chunks. The content similar to a user's question is retrieved using the semantic search library FAISS [28,29].

Experiment

The project has been carried out exploiting the computational resources of the supercomputer LEONARDO, hosted by CINECA. Each node in the booster partition is equipped with four NVidia A100 SXM6 64GB GPUs and a single 32-cores Intel Ice Lake CPU.

For all models, only data parallelism has been employed, given the fact that all these models could adequately fit in the VRAM of the GPUs at our disposal. For the same reason, LLaMAntino-2-7b-hf-ITA and Mistral-7B-v0.1 have not been quantized during domain adaptation and instruction fine-tuning, opting to preserve the weights' precision. Mixtral-8x7B-Instruct-v0.1 underwent 4-bit quantization instead [30], due to its size. For domain adaptation and instruction fine-tuning, we applied LoRA adapters on Q, K, V layers of the models [15]. The training procedure for the models under study were the following:

• pre-trained LLMs causal Language Modelling on the legal text chunks. This has been performed on LLaMAntino-2-7b-hf-ITA and Mistral-7B-v0.

Results

To evaluate the quality and proceed to select the candidate for instruction-based fine-tuning, all domainadapted models were evaluated using the perplexity metric (PPL) on an held-out evaluation dataset based on laws.

The metric is reported in Table 1.

Three different domain experts (lawmakers of Assemblea Legislativa) were asked to evaluate the answers generated by the final instruction fine-tuned models to a set of 25 questions. The qualitative analysis of the experts reported that, in general, the answers provided by the LLaMAntino-based model were considered too short and dry. The answers provided by the Mixtral-based model were considered the most complete, clear and satisfactory in terms of quality of the used specific words. Below we report an example of answers provided by the different final models on a given question. For context, we also include the answer of chatGPT (3.5) to the same question.

• Question: Sul tema della partecipazione, quali leggi sono state fatte in Emilia-Romagna? • Answer of Mixtral-8x7B-Instruct-v0.1 fine-tuned:

La prima legge regionale approvata in tema di partecipazione è la legge regionale 9 febbraio 2010, n.

Conclusions

We explored different approaches to adapt open-source LLMs for question-answering on the Emilia-Romagna law corpus. We adapted the different LLMs on a corpus composed of the Emilia-Romagna regional laws and the relative implementing acts, and we further refined the domain-adapted models on a custom QA dataset provided by domain experts. Finally, we exploited RAG to enrich the user's question with relevant contextual information extracted from the law database. We experimented with different open-source LLMs, such as Mistral-7B-v0.1, LLaMAntino-2-7b-hf-ITA, Mixtral-8x7B-Instruct-v0.1. Our results show that domain-adapted LLMs that are able to answer specific domain questions can be a helpful tool to support decisionmaking in specialized fields such as the legal domain, that often need to retrieve exact, concise and easy-tounderstand information from large and unstructured data sources.

Given the scope and the length of the project, several improvements to the workflow are foreseen in the near future, as well as the possibility to test with more pre-trained open source models, for example new Italiannative models that will be developed in the near future and domain adaptation of Mixture-of-Experts models (such as Mixtral-8x7B-v0.1). Our future work will also focus on further improving the retrieval module with better embedding models, and on applying more powerful techniques to train the LLMs, such as Direct Preference Optimization (DPO, [31]) and Reinforcement Learning from Human Feedback (RLHF, [32]).