The increasing development of bigger and bigger Large Language Models (LLM), reaching 70B parameters as for Meta LLMs (Llama 2 [1] and Llama 3 [2]) and more as for OpenAI ones (GPT-3 [3] and GPT-4 [4] 1 ), requires a significant computational resources for training, fine-tuning or inference. OpenAI models are accessible only upon payment via OpenAI API and cannot be downloaded in any way, while the open-source models by Meta are available also in the 8B and 13B parameters versions, and they can either be fine-tuned via Parameter-Efficient Fine-Tuning techniques (PEFT) [5] such as LoRA [6], or they can make direct inference using a 8-bit quantization [7] keeping the computational resources relatively small.
The availability of open-source small-size LLMs is crucial for developing Natural Language Process (NLP) applications that leverage a fine-tuning procedure over a specific domain or language, as for Anita [8], an Italian 8B adaptation of Llama 3. Nevertheless, GPT and Llama models cannot be considered as truly open-source since their training data set is not available and, as for GPT models, and also their actual architecture is not accessible. Minerva [9] model, on the other side, is an Italian and English LLM whose architecture, weights, and training data are accessible, but it
The increasing interest in developing Language Models (LM) for the Italian language, starts when BERT [12] was first released and adapted models, such as AlBERTo [13] were developed. After ChatGPT was made public [3,4], an increasing interest in developing and using LLMs, and in generative AI based on decoder-only model, was crucial, also for the Italian NLP community, thus leading to the development of foundational models based on Llama 2 [1] and Llama 3 [2]. Among those models, LLaMantino (chat version) [14] and Fauno [15], are based on Llama 2 fine-tuned for chat purposes, while Camoscio [16] and Anita [8] are a fine-tuned Italian version of the instruct version of Llama 2 and Llama 3, respectively. RAG is used in developing chat-bots which are grounded in various domains where the models need to be deeply guided in generation to avoid hallucination in their answers. Various examples can be found in the educational domain as for AI4LA [17], an assistant to students with Specific Learning Disorders (SLDs) like Dyslexia, Dysorthographia, and Dyscalculia, or as assistant providing information about restaurant industry [18] or as chat-bot for Frequently Asked Questions (FAQ) [19]. Also chat-bots for the Italian language were implemented for real-wold applications, namely as assistant for Italian Funding Application [20], or in the medical domain [21] or in industrial context [22]. The aforementioned works share the same architecture with the one we used to implement our model. In contrast with them, we decided to stress capabilities of open-source LLMs and do not rely on GPT-based models, that are used as baseline reference for text generation (gpt-3.5-turbo) and as an external judge to evaluate performances of the other models (gpt-4.5-turbo).
Open Unipa-GPT relies on two main components as it is shown in Figure 1 that is the Retriever and the Generator. In the following, the two components are detailed.
The Retriever is made up of a vector database built using the LangChain framework2 , which makes use of the Facebook AI Similarity Search (FAISS) library [23]. The vector database is filled with the documents belonging to the unipa-corpus (Appendix A), that are divided into 1K token chunks with an overlap of 50 tokens. Split documents are then processed by a LLM (the Embedding LLM) to generate the corresponding embedding, and store them in the vector database. Different LLMs were used for embedding generation: we selected the best models according to the Massive Text Embedding Benchmark (MTEB) [24] for Information Retrieval3 . We selected only models that explicitly state that they were trained and tested also with Italian data. In the end, we selected the following models: BGE-M3 (BGE) [25], E5-mistral-7b-instruct (E5-mistral) [26], sentence-bert-baseitalian-xxl-uncased 4 (BERT-it) and Multilingual-E5-large-instruct (m-E5) [27] .
A vector database was built for each model, and their corresponding embedding spaces were compared to each other and with text-embedding-ada-002, the embedding model from OpenAI, to asses their retrieval performances (Section 5).
The Generator uses the following Italian isntruction prompt to answer to user questions: Both the question and the related relevant context are passed as input to the model, along with the prompt. As regards the Generator LLM, we used Transformer-based models [28]. We choose not to use LLMs based on Llama 2 and deeply focused our work towards the most recent models, covering both Llama-and Mistral-based architectures. In particular, Llama-3-8B-instruct [2] was used along with its adapted version for Italian, Anita-8B [8], and Minerva-3B [9], which is a Mistral-based architecture [29]. All the generation LLMs were evaluated both in their base version and in the instruction-tuned one. The last ones were obtained via a three-epochs fine-tuning procedure with the Alpaca-LoRA [6] strategy testing the Alpaca-LoRA hyper-parameters 5 for both 20 and 50 epochs. In the generation phase, models were asked to output at most 256 tokens. We manually generated a small set of Question-Answer (QA) pairs for evaluation starting from the real questions issued by the public during the 2023 SHARPER European Researchers' Night where Unipa-GPT was demonstrated. The procedure for building these QA pairs is reported in Section 4. We developed the entire system on a server with 2 Intel(R) Xeon(R) 6248R CPUs, 384 GB RAM, and two 48 GB NVIDIA RTX 6000 Ada Generation GPUs.
The Italian documents data set built for Unipa-GPT is called unipa-corpus [10], and it has been generated from scraping either HTML pages or PDF documents that are publicly available on the website of the University of Palermo, and it includes information about all the available Bachelor/Master degree courses in the academic year 2023/2024 along with practical information for future students, e.g. how to pay taxes, the enrollment procedure, and the related deadlines. Starting from this data set, a QA data set was created with a semisupervised procedure to allow instruction-tuning over general-purpose LLMs. Further information about the unipa-corpus is reported in Appendix A.
As already mentioned The original Unipa-GPT was available for public unsupervised QA during the European Researchers' Night in 2023, where a total of 165 questions was collected, along with feedback of users. On average, an interaction with the chat-bot was two questions long, and we collected qualitative evaluation of the user experience through a suitable questionnaire people were requested to fill on line just after having chatted with Unipa-GPT. Questionnaires were further analyzed, and resulted in a general positive evaluation of the system's performances by the majority of the users, which were mostly University students.
To generate the golden QA pairs used to assess the different performances of each generator LLM, we devised six typologies by the direct inspection of collected questions. Particularly we groupte questions in Generic Information, Courses' Information, Other University-related, Services and Structures, Taxes and Scholarships, University Environment, and Off-topic. Next, we picked one question per typology, discarding the Off-topic ones, and a golden answer was manually built for each of them by leveraging the actual relevant documents contained in the corpus, thus marking them as golden documents. Note that if an answer can be elicited by multiple documents, all of them have been marked as golden. The detailed list of the Italian QA pairs is reported in Appendix B in Table 4, while the English version is reported in Table 5. Note that the English version is reported here for full readability purposes, while only Italian data were used for evaluation.
The proposed model is intended to work in an open QA context, where correct answers are not known, thus, after a previous phase of qualitative evaluation [10] as in [17,20,21,22], we opted for a quantitative analysis, relying on the small QA data set described in Section 4 to evaluate the performances against a set of golden labels in terms of both retrieval and answering capabilities [30,19,18].
For each QA test pair, we retrieved the four most relevant documents from each vector database related to one of the open Embedding LLMs under investiga- tion. Then we scored the retrieved documents in terms of their context relevancy with respect to the provided question using the RAGAS framework [31] that exploits gpt-4-turbo for the evaluation. Results are reported in Table 1, and they include also the performances of the original vector database using OpenAI embeddings (text-embedding-ada-002, referred as open-ai-ada).
The overall scores are not so high, and also the highest relevancy do not always correspond to the golden document used for generating the corresponding answer. In Table 1 the underlined values are the ones associated with golden documents, while the bold ones are the highest RAGAS values. A model is considered to perform correctly if the highest context relevancy score is assigned to one of the golden documents. This evaluation procedure led to select E5-mistral as the best performing Embeddings LLM among the ones we investigated. Superior performances of E5-mistral are also confirmed by a deep analysis on the embeddings space by means of two different clustering procedures. We clustered the embeddings generated by each LLM starting from the documents belonging to both sections Educational Offer and Future Students of the UniPA website. The firs group of documents is the list of all the available courses at the University, while the second group contains useful information for future students who want to enroll in a degree course. We clustered the embedding spaces according to the either the course degree typology (bachelor/master degree) or the Department where a degree course is affiliated to. Quantitative measures of the clustering goodness are reported in Table 2, where the Silhouette Coefficients [32] have been computed for each model, and again E5-mistral is the best performing one. In Appendix C, we report the scatter plots of the embedding spaces for each Embeddings LLM (Figure 3 and Figure 4 ). Plots have been obtained through a 2D dimensionality reduction using t-SNE [33].
We used the six QA test pairs to obtain also a quantitative evaluation of the correctness of the answers provided by all the Generation LLMs under investigation. Comparison was carried out against both the golden answers and the ones generated via gpt-3.5-turbo (GPT) in the original Unipa-GPT set up. The proposed evaluation task, can be regarded as an open QA one where, despite a golden answer is provided for a given question, diverse correct answers can be proposed with different linguistic nuances, according to Italian diaphasic variation [34]. To evaluate both strict and light correctness of the generated answers, we employed traditional QA metrics such as BLEU [35] (Figure 2.a) and ROGUE-L score [36] (Figure 2.b) and novel metrics leveraging the RAGAS framework [31] to evaluate Faithfulness (Figure 2.c) and Correctness (Figure 2.d) of the generated output. Such measures request an external LLM acting as a "judge", and we we used gpt-4-turbo in this respect. More specifically, Faithfulness measures the factual consistency of the generated answer against the given context, while Correctness involves gauging the accuracy of the generated answer when compared to the ground truth. Both metrics range from 0 to 1 and better performances are associated with higher scores.
Both BLEU and ROUGE scores are generally low, but we assume that this is mainly related to the fact that an exact match cannot be reached between the golden answer and the generated one, and a more semantically comparison should be taken into account. Overall, answers generated by gpt-3.5-turbo can be considered as the best ones as they attain highest values. By contrast, fine-tuning did not provided a desired improvement in the open-source models: all BLEU scores are almost zero, except for Anita-8B . ROUGE scores are higher than the corresponding BLEU ones, and again the base ver-
Silhouette Coefficients for each Embedding LLM with reference to the two proposed clustering schemes, that is the degree courses typology and their affiliation to a particular Department. sion of each LLM performes better than the fine-tuned ones. Generally speaking, Anita-8B and Llama-3-8Binstruct outperform Minerva, since both reach comparable scores, but we assume that the tailored Italian fine-tuning over Llama-3 to obtain Anita-8B was crucial to make it the best performing open-source model during this first automatic evaluation phase. gpt-3.5-turbo exhibits the best Faithfulness scores despite being surpassed by Anita-8B in question Q2, and also these results confirm the previous considerations about BLEU scores. Something changes in evaluating models in terms of their Correctness: in this case gpt-3.5-turbo is the best model in three answers out of six, followed by Anita-8B (two best results) and Minerva-3B-20 (one best result). We are aware that gpt-based evaluation may lead to a preference over GPT models themselves, but gpt-4-turbo was the only high quality generative model we had access to at the time of making the experiments.
Overall results confirm that a (moderate) fine-tuning is not significantly beneficial in terms of performance increase for any model and, even if it does not reach the same performances, Anita-8B seems to be the most valuable alternative to GPT. A manual inspection of the generated answers, outlines a common issue related to the tokenization of the generated output: despite of its semantic correctness, the generated text is outputted as a unique word without any spaces, as Glielezionidelcorsosaracondottoattraversounaprocesso in Llama-3-8B-instruct, or it is over-splitted as e-domandre d'i-s-c-r-i-z-ion-e-per-l-A.-A.-2023-/--cor-so-n-d'-l-a-u-re-'-(M-agg
in Anita-8B. These errors make the models not suitable for human interaction, since it is not possible read the generated answers. We argue that a deeper analysis on the tokenizer that has been used and, a hyper-parametrs tuning in the generator, may lead to an increase of performances. Models tend also to answer in other languages as * La durada édié depresso àdue años, * Accesso libre! * Dipartment of Physics & Chemistry "Emilo Segré" Codice course : 21915 in Llama-3-8B-instruct. We argue that this trouble can be related to the memory of multi-lingual models that uses texts also in French and Spanish despite the Italian fine-tuning. It is worth noticing that those languages are linguistically close to Italian and together belong to the Romance Languages [37]. Thus, even if the output has to be considered wrong, a linguistic connection can be highlighted.
The most unsatisfactory results are reported for Minerva-3B: the model does not generate any answer related to the given question, and it seems that answers where generated with samples from model's training set. As stated before, a tuning of the generator hyperparameters may help in this case.
Despite the promising results, in some cases answers by both Anita-8B and Llama-3-8B-instruct are not good from a grammatical point of view, since they are full of mistakes, thus making them not yet ready to be used in real-world applications compared to OpenAI's ones.
In this paper we presented Open Unipa-GPT, a virtual assistant, which is based solely on open-source LLMs, and uses a RAG approach to answer Italian universityrelated questions from secondary school students. The main intent of the presented research was setting up a sort of framework to test open-source small size LLMs, with either moderate or no fine-tuning at all, to be used for generating the embeddings and/or as text generation front-end in a RAG set up.
Our study led us to devise E5-mistral-7b-instruct as a valuable open-source alternative to OpenAI's embeddings, while none of the considered models attain a generation performance comparable to gpt-3.5-turbo, even after a fine-tuning procedure. The most promising Generation LLM, when plunged in our architecture, appears to be Anita-8B, but it still shows some issues related to both the tokenization and the grammatical correctness of the output. We are currently working to deep exploration of different fine-tuning approaches along with the use of huge size open-source LLMs for text generation.
unipa-corpus [10] is a collection of Italian documents that were retrieved directly from the website of the University of Palermo in Semptember 2023. The corpus is divided in two main sections, namely Education, that groups the available bachelor and master degree courses, and Future Students where important information about taxes payment and enrollment procedure are reported. For fine-tuning purposes, a semi-automatic procedure, involving gpt-3.5-turbo [3], was implemented to build a QA dataset. In Table 3 are reported the statistics of unipa-corpus.