This paper illustrates the implementation of Open Unipa-GPT, an open-source version of the Unipa-GPT chat-bot that leverages open-source Large Language Models for embeddings and text generation. The system relies on a Retrieval Augmented Generation approach, thus mitigating hallucination errors in the generation phase. A detailed comparison between diferent models is reported to illustrate their performance as regards embedding generation, retrieval, and text generation. In the last case, models were tested in a simple inference setup after a fine-tuning procedure. Experiments demonstrate that an open-source LLMs can be eficiently used for embedding generation, but none of the models does reach the performances obtained by closed models, such as gpt-3.5-turbo in generating answers. Corpora and code are available on GitHub1 Workshop Proceedings available also in the 8B and 13B parameters versions, in our original version, namely text-embedding-adacial for developing Natural Language Process (NLP) ap- and related results are reported in Section 5. Finally,
CEUR
ceur-ws.org
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Anita [
OpenAI ones (GPT-3 [
they can make direct inference using a 8-bit quantization
[
plications that leverage a fine-tuning procedure over a
specific domain or language, as for Anita [
Nevertheless, GPT and Llama models cannot be
considered as truly open-source since their training data set
is not available and, as for GPT models, and also their
actual architecture is not accessible. Minerva [
(chat version) [14] and Fauno [15], are based on Llama
version of Llama 2 and Llama 3, respectively. the other side, is an Italian and English LLM whose archi- first released and adapted models, such as AlBERTo [ 13] CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, in generative AI based on decoder-only model, was cru
RAG is used in developing chat-bots which are 1K token chunks with an overlap of 50 tokens. Split docgrounded in various domains where the models need uments are then processed by a LLM (the Embedding to be deeply guided in generation to avoid hallucination LLM ) to generate the corresponding embedding, and in their answers. Various examples can be found in the store them in the vector database. Diferent LLMs were educational domain as for AI4LA [17], an assistant to used for embedding generation: we selected the best modstudents with Specific Learning Disorders (SLDs) like els according to the Massive Text Embedding Benchmark Dyslexia, Dysorthographia, and Dyscalculia, or as as- (MTEB) [24] for Information Retrieval3. We selected only sistant providing information about restaurant industry models that explicitly state that they were trained and [18] or as chat-bot for Frequently Asked Questions (FAQ) tested also with Italian data. In the end, we selected [19]. Also chat-bots for the Italian language were imple- the following models: BGE-M3 (BGE) [25], E5-mistralmented for real-wold applications, namely as assistant 7b-instruct (E5-mistral) [26], sentence-bert-basefor Italian Funding Application [20], or in the medical italian-xxl-uncased4 (BERT-it) and Multilingualdomain [21] or in industrial context [22]. The aforemen- E5-large-instruct (m-E5) [27] . tioned works share the same architecture with the one A vector database was built for each model, and their we used to implement our model. In contrast with them, corresponding embedding spaces were compared to each we decided to stress capabilities of open-source LLMs other and with text-embedding-ada-002, the embedand do not rely on GPT-based models, that are used as ding model from OpenAI, to asses their retrieval perforbaseline reference for text generation (gpt-3.5-turbo) mances (Section 5). and as an external judge to evaluate performances of the other models (gpt-4.5-turbo).
shown in Figure 1 that is the Retriever and the Generator. In the following, the two components are detailed.
ing the LangChain framework2, which makes use of the Facebook AI Similarity Search (FAISS) library [23]. The vector database is filled with the documents belonging to the unipa-corpus (Appendix A), that are divided into
Sei Unipa-GPT, chatbot e assistente virtuale
dell’Università degli Studi di Palermo che
risponde cordialmente e in forma
colloquiale. Ai saluti, rispondi salutando e
presentandoti. Ricordati che il rettore
dell’Università è il professore Massimo Midiri. Se la
domanda riguarda l’università degli studi di
Palermo, rispondi in base alle informazioni
e riporta i link ad esse associati; Se non
3as in https://huggingface.co/spaces/mteb/leader-board in June 2024
4https://huggingface.co/nickprock/
sentence-bert-base-italian-xxl-uncased
The proposed model is intended to work in an open QA
context, where correct answers are not known, thus, after
a previous phase of qualitative evaluation [
For each QA test pair, we retrieved the four most relevant documents from each vector database related to one of the open Embedding LLMs under investiga
called unipa-corpus [
tion. Then we scored the retrieved documents in terms dimensionality reduction using t-SNE [33]. of their context relevancy with respect to the provided We used the six QA test pairs to obtain also a quantitaquestion using the RAGAS framework [31] that exploits tive evaluation of the correctness of the answers provided gpt-4-turbo for the evaluation. Results are reported by all the Generation LLMs under investigation. Comin Table 1, and they include also the performances of parison was carried out against both the golden answers the original vector database using OpenAI embeddings and the ones generated via gpt-3.5-turbo (GPT) in the (text-embedding-ada-002, referred as open-ai-ada). original Unipa-GPT set up. The proposed evaluation The overall scores are not so high, and also the highest task, can be regarded as an open QA one where, despite relevancy do not always correspond to the golden docu- a golden answer is provided for a given question, diverse ment used for generating the corresponding answer. In correct answers can be proposed with diferent linguistic Table 1 the underlined values are the ones associated with nuances, according to Italian diaphasic variation [34]. To golden documents, while the bold ones are the highest evaluate both strict and light correctness of the generated RAGAS values. A model is considered to perform cor- answers, we employed traditional QA metrics such as rectly if the highest context relevancy score is assigned BLEU [35] (Figure 2.a) and ROGUE-L score [36] (Figure to one of the golden documents. This evaluation pro- 2.b) and novel metrics leveraging the RAGAS framework cedure led to select E5-mistral as the best performing [31] to evaluate Faithfulness (Figure 2.c) and Correctness Embeddings LLM among the ones we investigated. (Figure 2.d) of the generated output. Such measures re
Superior performances of E5-mistral are also con- quest an external LLM acting as a “judge”, and we we used ifrmed by a deep analysis on the embeddings space by gpt-4-turbo in this respect. More specifically, Faithfulmeans of two diferent clustering procedures. We clus- ness measures the factual consistency of the generated tered the embeddings generated by each LLM starting answer against the given context, while Correctness infrom the documents belonging to both sections Educa- volves gauging the accuracy of the generated answer tional Ofer and Future Students of the UniPA website. when compared to the ground truth. Both metrics range The firs group of documents is the list of all the available from 0 to 1 and better performances are associated with courses at the University, while the second group con- higher scores. tains useful information for future students who want to Both BLEU and ROUGE scores are generally low, but enroll in a degree course. We clustered the embedding we assume that this is mainly related to the fact that spaces according to the either the course degree typol- an exact match cannot be reached between the golden ogy (bachelor/master degree) or the Department where answer and the generated one, and a more semantically a degree course is afiliated to. Quantitative measures of comparison should be taken into account. Overall, anthe clustering goodness are reported in Table 2, where swers generated by gpt-3.5-turbo can be considered as the Silhouette Coeficients [ 32] have been computed for the best ones as they attain highest values. By contrast, each model, and again E5-mistral is the best performing ifne-tuning did not provided a desired improvement in one. In Appendix C, we report the scatter plots of the the open-source models: all BLEU scores are almost zero, embedding spaces for each Embeddings LLM (Figure 3 except for Anita-8B . ROUGE scores are higher than and Figure 4 ). Plots have been obtained through a 2D the corresponding BLEU ones, and again the base version of each LLM performes better than the fine-tuned is not significantly beneficial in terms of performance ones. Generally speaking, Anita-8B and Llama-3-8B- increase for any model and, even if it does not reach instruct outperform Minerva, since both reach com- the same performances, Anita-8B seems to be the most parable scores, but we assume that the tailored Italian valuable alternative to GPT. ifne-tuning over Llama-3 to obtain Anita-8B was crucial A manual inspection of the generated answers, outto make it the best performing open-source model during lines a common issue related to the tokenization of the this first automatic evaluation phase. generated output: despite of its semantic correctness, the gpt-3.5-turbo exhibits the best Faithfulness scores generated text is outputted as a unique word without any despite being surpassed by Anita-8B in question Q2, and spaces, as also these results confirm the previous considerations about BLEU scores. Something changes in evaluating Glielezionidelcorsosaracondottoattravermodels in terms of their Correctness: in this case gpt- sounaprocesso 3.5-turbo is the best model in three answers out of six, in Llama-3-8B-instruct, or it is over-splitted as followed by Anita-8B (two best results) and Minerva3B-20 (one best result). We are aware that gpt-based e-domandre d’i-s-c-r-i-z-ion-e-per-l-A.evaluation may lead to a preference over GPT models A.–2023–/—-cor-so-n-d’-l-a-u-re-’-(M-agthemselves, but gpt-4-turbo was the only high quality g-is-t-ra-le)–a-dd-ac-ce-o-lib-ro generative model we had access to at the time of making the experiments.
Overall results confirm that a (moderate) fine-tuning in Anita-8B. These errors make the models not suitable for human interaction, since it is not possible read the generated answers. We argue that a deeper analysis on the tokenizer that has been used and, a hyper-parametrs tuning in the generator, may lead to an increase of performances. Models tend also to answer in other languages as * La durada édié depresso àdue años, * Accesso libre! * Dipartment of Physics & Chemistry “Emilo Segré” Codice course : 21915 in Llama-3-8B-instruct. We argue that this trouble can be related to the memory of multi-lingual models that uses texts also in French and Spanish despite the Italian ifne-tuning. It is worth noticing that those languages are linguistically close to Italian and together belong to the Romance Languages [37]. Thus, even if the output has to be considered wrong, a linguistic connection can be highlighted.
The most unsatisfactory results are reported for Minerva-3B: the model does not generate any answer related to the given question, and it seems that answers where generated with samples from model’s training set. As stated before, a tuning of the generator hyperparameters may help in this case.
Despite the promising results, in some cases answers by both Anita-8B and Llama-3-8B-instruct are not good from a grammatical point of view, since they are full of mistakes, thus making them not yet ready to be used in real-world applications compared to OpenAI’s ones.
In this paper we presented Open Unipa-GPT, a virtual assistant, which is based solely on open-source LLMs, and uses a RAG approach to answer Italian universityrelated questions from secondary school students. The main intent of the presented research was setting up a sort of framework to test open-source small size LLMs, with either moderate or no fine-tuning at all, to be used for generating the embeddings and/or as text generation front-end in a RAG set up.
Our study led us to devise E5-mistral-7b-instruct as a valuable open-source alternative to OpenAI’s embeddings, while none of the considered models attain a generation performance comparable to gpt-3.5-turbo, even after a fine-tuning procedure. The most promising Generation LLM, when plunged in our architecture, appears to be Anita-8B, but it still shows some issues related to both the tokenization and the grammatical correctness of the output. We are currently working to deep exploration of diferent fine-tuning approaches along with the use of huge size open-source LLMs for text generation. https://aclanthology.org/N19-1423. doi:10.18653/ [25] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, v1/N19- 1423. Z. Liu, Bge m3-embedding: Multi-lingual, [13] M. Polignano, P. Basile, M. Degemmis, G. Semeraro, multi-functionality, multi-granularity text emV. Basile, Alberto: Italian bert language under- beddings through self-knowledge distillation, standing model for nlp challenging tasks based on 2024. URL: https://arxiv.org/abs/2402.03216. tweets, in: Italian Conference on Computational arXiv:2402.03216.
Linguistics, 2019. URL: https://api.semanticscholar. [26] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, org/CorpusID:204914950. F. Wei, Improving text embeddings with large lan[14] P. Basile, E. Musacchio, M. Polignano, L. Siciliani, guage models, arXiv preprint arXiv:2401.00368 G. Fiameni, G. Semeraro, Llamantino: Llama 2 mod- (2023). els for efective text generation in italian language, [27] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, 2023. arXiv:2312.09993. F. Wei, Multilingual e5 text embeddings: A techni[15] A. Bacciu, G. Trappolini, A. Santilli, E. Rodolà, F. Sil- cal report, arXiv preprint arXiv:2402.05672 (2024). vestri, Fauno: The italian large language model [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, that will leave you senza parole!, arXiv preprint L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, AtarXiv:2306.14457 (2023). tention is all you need, Advances in neural infor[16] A. Santilli, E. Rodolà, Camoscio: an italian mation processing systems 30 (2017).
instruction-tuned llama, 2023. arXiv:2307.16456. [29] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bam[17] S. D’Urso, F. Sciarrone, Ai4la: An intelligent chat- ford, D. S. Chaplot, D. de las Casas, F. Bressand, bot for supporting students with dyslexia, based on G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.generative ai, in: A. Sifaleras, F. Lin (Eds.), Gen- A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, erative Intelligence and Intelligent Tutoring Sys- T. Lacroix, W. E. Sayed, Mistral 7b, 2023. URL: https: tems, Springer Nature Switzerland, Cham, 2024, pp. //arxiv.org/abs/2310.06825. arXiv:2310.06825. 369–377. [30] S. Vidivelli, M. Ramachandran, A. Dharunbal[18] V. Bhat, D. Sree, J. Cheerla, N. Mathew, G. LIu, J. Gao, aji, Eficiency-driven custom chatbot developRetrieval augmented generation (rag) based restau- ment: Unleashing langchain, rag, and performancerant chatbot with ai testability, 2024. optimized llm fusion., Computers, Materials & Con[19] M. Kulkarni, P. Tangarajan, K. Kim, A. Trivedi, Re- tinua 80 (2024).
inforcement learning for optimizing rag for do- [31] S. Es, J. James, L. Espinosa-Anke, S. Schockaert, Ramain chatbots, 2024. URL: https://arxiv.org/abs/ gas: Automated evaluation of retrieval augmented 2401.06800. arXiv:2401.06800. generation, arXiv preprint arXiv:2309.15217 (2023). [20] T. Boccato, M. Ferrante, N. Toschi, Two-phase rag- [32] P. J. Rousseeuw, Silhouettes: a graphical aid to based chatbot for italian funding application assis- the interpretation and validation of cluster analysis, tance, 2024. Journal of computational and applied mathematics [21] S. Ghanbari Haez, M. Segala, P. Bellan, S. Mag- 20 (1987) 53–65.
nolini, L. Sanna, M. Consolandi, M. Dragoni, A [33] L. Van der Maaten, G. Hinton, Visualizing data retrieval-augmented generation strategy to en- using t-sne., Journal of machine learning research hance medical chatbot reliability, in: J. Finkelstein, 9 (2008).
R. Moskovitch, E. Parimbelli (Eds.), Artificial Intel- [34] G. Berruto, Variazione diafasica, 2011. ligence in Medicine, Springer Nature Switzerland, URL: https://www.treccani.it/enciclopedia/ Cham, 2024, pp. 213–223. variazione-diafasica_(Enciclopedia-dell'Italiano)/. [22] R. Figliè, T. Turchi, G. Baldi, D. Mazzei, Towards an [35] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a llm-based intelligent assistant for industry 5.0, in: method for automatic evaluation of machine transProceedings of the 1st International Workshop on lation, in: Proceedings of the 40th annual meeting Designing and Building Hybrid Human–AI Systems of the Association for Computational Linguistics, (SYNERGY 2024), volume 3701, 2024. URL: https: 2002, pp. 311–318.
//ceur-ws.org/Vol-3701/paper7.pdf. [36] C.-Y. Lin, Rouge: A package for automatic eval[23] J. Johnson, M. Douze, H. Jégou, Billion-scale simi- uation of summaries, in: Text summarization larity search with GPUs, IEEE Transactions on Big branches out, 2004, pp. 74–81.
Data 7 (2019) 535–547. [37] T. Alkire, C. Rosen, Romance languages: A histori[24] N. Muennighof, N. Tazi, L. Magne, N. Reimers, cal introduction, Cambridge University Press, 2010.
Mteb: Massive text embedding benchmark, arXiv preprint arXiv:2210.07316 (2022). URL: https: //arxiv.org/abs/2210.07316. doi:10.48550/ARXIV.
2210.07316.
A. unipa-corpus details
unipa-corpus [
For fine-tuning purposes, a semi-automatic procedure,
involving gpt-3.5-turbo [
Answers Il professore di Intelligenza Artificiale 1 del corso di Laurea Magistrale in Ingegneria Informatica è il professore Gaglio e l’insegnamento verrà erogato durante il primo semestre. Per maggiori informazioni vai su http://www.unipa.it/struttura.html?id=721 La presentazione delle domande di iscrizione per l’Anno Accademico 2023/2024 varia in base alla tipologia di corso. Per i corsi di studio triennali e magistrali a ciclo unico, la domanda può essere presentata dall’1 agosto al 30 settembre 2023 , mentre per i corsi magistrali, dall’1 agosto al 30 novembre 2023, in entrambi i casi, è richiesto il pagamento della prima rata delle tasse universitarie. Per i corsi di studio ad accesso programmato locale, con prenotazione online o ad accesso programmato nazionale, le immatricolazioni vanno efettuate entro i termini previsti dal relativi bandi di concorso.
Il corso di laurea magistrale in Chimica presso l’Università degli Studi di Palermo, ha sede a Palermo e ha una durata di 2 anni. L’obiettivo del corso è la formazione di laureati con una solida preparazione di base, che li ponga in grado di afrontare con competenza ed autonomia sia attività professionali sia quelle di ricerca accademica ed industriale. Per questo motivo il corso di studio è progettato con lo scopo di completare la cultura scientifica nelle discipline chimiche fondamentali e di introdurre conoscenze più avanzate. Quindi lo studente può completare il proprio piano di studi, attenzionando gli insegnamenti che meglio si adattano ai propri interessi specifici, come la chimica dei materiali, dell’ambiente e dei beni culturali, la chimica supramolecolare e la didattica della chimica. Per maggiori informazioni vai su http://www.unipa.it/struttura.html?id=766 Ciao! Il corso di Laurea in Consulente Giuridico d’Impresa sembra proprio fare al caso tuo! Il corso di laurea ha sede a Trapani e ha una durata di 3 anni e ti fornirà le conoscenze in ambito giuridico ed economico-aziendalistico. Una volta terminato il percorso di studio potrai svolgere attività interdisciplinari che richiedono competenze giuridiche, aziendalistiche e organizzativo-gestionali. Per maggiori informazioni vai su http://www.unipa.it/struttura.htmlid=1557 È possibile recarsi in segreteria il lunedì, mercoledì e venerdì dalle 10.00 alle 12.00, martedì e giovedì dalle 15.00 alle 17.00 . Puoi prenotare il tuo turno attraverso la App SolariQ. Per maggiori informazioni vai su https://www.unipa.it/servizi/segreterie/ Il pagamento delle tasse deve essere efettuato esclusivamente mediante sistema PAgoPA (Pagamenti della Pubblica Amministrazione). Dopo aver compilato la pratica online, è possibile pagare direttamente online con il sistema PAgoPA o stampare il bollettino e pagare presso tabaccai convenzionati o ricevitorie abilitate PAgoPA. Ulteriori informazioni sul pagamento via PAgoPA sono reperibili qui https://immaweb.unipa.it/ immaweb/public/pagamenti.seam, mentre è disponibile il Regolamento in materia di contribuzione studentesca https://www.unipa.it/servizi/segreterie/ .content/documenti/regolamenti_calendari/2023/5105144def_regolamento-contribuzione–studentesca-2023—24-2.pdf
Answers The Artificial Intelligence 1 professor for the Computer Engineering Master degree course is Professor Gaglio and it will be delivered during the first semester. For more information go to http://www.unipa.it/struttura.html?id=721 The submission of applications for the Academic Year 2023/2024 varies according to the type of course. For three-year and single-cycle master’s degree courses, applications can be submitted from 1 August to 30 September 2023, while for master’s degree courses, from 1 August to 30 November 2023; in both cases, payment of the first instalment of tuition fees is required. For courses with local programmed access, with online booking or national programmed access, enrolment must be carried out by the deadlines set out in the corresponding calls for application.
The Master’s degree course in Chemistry at the University of Palermo is based in Palermo and lasts 2 years. The aim of the course is to train graduates with a good background, enabling them to deal competently and independently with both professional activities and academic and industrial research. For this reason, the course is designed with the aim of completing the scientific culture in the fundamental chemical disciplines and introducing more advanced knowledge. Therefore, students can complete their study plan by focusing on the subjects that best suit their specific interests, such as the chemistry of materials, the environment and cultural heritage, supramolecular chemistry and the didactics of chemistry. For more information go to http://www.unipa.it/struttura.html?id=766 Hi! The Bachelor of Business Law Consultant programme sounds like it could be just the thing for you! The degree course is based in Trapani and lasts 3 years and will provide you with knowledge in the fields of law and business economics. Once you have completed the course you will be able to carry out interdisciplinary activities requiring legal, business and organisational-managerial skills. For more information go to http://www.unipa.it/struttura.html?id=1557 You can go to the secretariat on Mondays, Wednesdays and Fridays from 10 a.m. to 12 noon, Tuesdays and Thursdays from 3 p.m. to 5 p.m. . You can book your appointment through the SolariQ App. For more information go to https://www.unipa.it/servizi/segreterie/ Fees must be paid exclusively through the PAgoPA (Public Administration Payments) system, which is accessed through the university portal. After completing the paperwork online, you can either pay directly online via the PAgoPA system or print out the payment slip and pay at a PAgoPA-enabled tax ofice. Further information on paying via PAgoPA can be found here https://immaweb.unipa.it/immaweb/public/pagamenti. seam, while the Student Contribution Regulations is available here https://www.unipa.it/servizi/segreterie/.content/documents/regulations_ calendars/2023/5105144-def_regulation-student-contribution-2023-24-2.pdf