The rapid progression of Generative Artificial Intelligence, together with researchers' increasing attention to the Public Administration (PA), opens up to novel cross-domain applications. Generative AI technologies like Large Language Models (LLMs) can expedite administrative purchasing processes and boost the transparency of the procurement life cycle. However, in a dynamic domain like the PA, updating LLMs training data can prove very prohibitive. Recently emerged Graph Retrieval-Augmented Generation (RAG) solves this data editing limitation, and tackles the lack of global information of traditional RAG techniques. Graph RAG leverages structural information across entities, enabling more comprehensive, context-aware responses. This paper illustrates a preliminary application of MicrGorsoaft'pshRAG in the PA domain, leveraging the latest Italian Public Contract Code corpus version. The experimental setting consists of an interface to let PA domain experts query the model about the Public Contract Code and evaluate the answers' correctness, completeness and fluency. Then, users filled out a satisfaction questionnaire to assess system usability and users' resistance to integrating this tool into their workflow. Results reveal a general users' satisfaction with the system: it achieves a System Usability Score of 82.19 and a Net Promoter Score of 25. Questions for assessing the correctness, completeness and fluency of the answers to users' queries achieve a mean score abo3v.7e0. Finally, results of the survey for assessing the users' resistance measured in terms of Perceived Value, Switching Benefit, Switching Cost, and Self-eficacy For Change - make clear that users consider this tool beneficial to their way of working.
The rapid surge of Generative AI has pervaded numerous domains, thanks to its wide range of applications. Among those domains, the Public Administration (PA) may greatly benefit from integrating Generative AI into the workflow. Even before the surge of Generative AI, researchers have shown a growing interest in Public E-procurem1e[1n,t2, 3, 4, 5, 6]. The goal of Public E-procurement is to automate a public procurement procedure to purchase goods, works or services. Such technologies boost and expedite administrative purchasing processes, expanding market player participation and preserving transparency of the procurement life cycle, hence clarifying and guaranteeing the accuracy of the necessary controls.
With the rise of Generative AI technologies – e.g., Large Language Models (LLMs) – this goal is becoming more achievable in a domain that heavily relies on textual documents. The integration of an LLM might be convenient for assisting both PA professionals and citizens in the decision-making and information access processes. For instance, the assistance of LLMs in the administrative environment may ease the task of retrieving in a single answer a piece of information that requires analysing a
ceur-ws.org
number sources2[
Despite their significant contribution, information encoded in LLMs is limited to the data they have been trained on, and this information can become obso7l,e8t].eIn[ a constantly changing domain like PA, where regulations, directives and guidelines are subject to frequent updates and corrections, these data-wise limitations are much more evident, and maintaining data pertinence can be restrictive.
To address these limitations, the introduction of non-parametric memorisation techniques – e.g.,
Retrieval-Augmented Generation (RA9G],) A[daptive Retrieva1l0[], Graph Retrieval-Augmented
Generation1[
However, traditional RAG techniques still face several limitations in real-world13s]c.eTnhaeriiros [
semantic similarity approach is not suitable for capturing the textual interconnections and relationa
knowledge. Excessively lengthy context in prompts can degrade the perfor1m4a]nncoet: i[ced that
a better performance is achieved when relevant information occurs at the beginning or end of the
input context, but worsens when relevant information is in the middle of long contexts. Lastly, vector
RAG techniques cannot grasp global information, so they cannot adequately perform Query-Focused
Summarisation (QFS) tasks, that saernese-making queries requiring a global comprehension of the data
[
This paper illustrates a preliminary experiment with Graph Retrieval-Augmented Generation in the Public Administration domain, leveraging the corpus of the Italian Public Contract Co2de. Section describes the data and methodology used, and Se3cttihoenexperimental setting and results. Finally, Section4 lays out the conclusions and presents some future works.
The corpus leveraged for this experiment isIttahliean Public Contract Code, last updated with the Legislative Decree 36/2022.3
The Italian Public Contract Code, or Tender Code, is a law issued by the Italian Republic to regulate public tenders and administrative concessions. It was instituted in 1924, and has undergone several changes since 1994. This regulatory text describes the procedures through which the Public Administration acquires goods and services, awards contracts, and grants concessions. When the public sector needs to meet its procurement requirements, it must act under the rules of public procurement, a fundamental principle for selecting the contractor. Therefore, the phases leading to the selection of the contractor are determined by administrative law under the jurisdiction of the administrative judge, while the contract signed with the contractor is ruled by civil law under the jurisdiction of the civi judge. 2Available in Italian hehrtetps://www.bosettiegatti.eu/info/norme/statali/2023_00,36in.htEmnglish herehttps://www. codiceappalti.it/Home/Legge/?legge=Italian_Procurement_Code_Decree_.36/2023
The Italian Public Contract Code consists of 229 articles, divided into 5 volumes, plus 37 attachments: I. General principles and provisions regarding the digitalization of public contracts, their planning, and design; II. Contracts for works, services, and supplies. It provides information on the relevant parties, particularly the contracting authorities on one side and economic operators on the other; III. Procurement in special sectors; IV. Public-private partnerships and concessions; V. Dispute management, the National Anti-Corruption Aut3,haonrditiyncludes final and transitional provisions.
With respect to former versions, the current Legislative Decree 36/2023 issued on M2a0r23ch 31 systematises numerous reform requests and amending decrees, in order to speed up procedures and address the needs arising from the COVID-19 pandemic. Furthermore, driven by projects related to the National Recovery and Resilience P4,liatnfosters transparency, digitalization of procedures, and their dematerialization and full traceability.
The Public Contract Code embodies the primary regulatory text a PA operator refers to when dealing with bid procedures, since it regulates their entire life-cycle, from the preparatory phase, to the participation requirements, until the definitive adjudication and the contract stipulation. Therefore, it is a fitting corpus for the preliminary experiment we intend to carry out, involving professionals from the PA domain.
GraphRAG5 [
You are a helpful assistant responsible for generating a comprehensive summary of the data provided below.
Conversely, below is the persona of the auto-tuned prompt:
You are an expert in public administration and e-procurement. You are skilled at analysing community structures and relationships, particularly in the context of digital procurement systems. You are adept at helping organizations understand the dynamics of their e-procurement communities, facilitating collaboration, and improving procurement processes. Using your expertise, you’re asked to generate a comprehensive summary of the data provided below. 3Autorità Nazionale Anticorruzione (ANAC) 4Piano Nazionale di Ripresa e Resilienza (PNRR) 5https://microsoft.github.io/graphrag/ 6https://microsoft.github.io/graphrag/prompt_tuning/auto_prompt_tuning/
To further exemplify, while in the default prompts the entity types to extorragcatniasraetion, geo, person, the entity types in the auto-tuned prompts inpculbuldiec administration, contract, project, service, subcontractor, authority, regulation.
An initial knowledge graph is created upon completion of the extraction step. The resulting graph of the Public Contract Code consists of 2,089 entities and 3,250 relations.
Clustering. Hierarchical clustering with Leiden techn2i1q]uies p[erformed on the knowledge graph, to detect community structures within the graph; entities in each cluster are distributed across diferent communities for a more detailed analysicso.mAmunity is a group made of densely intra-connected nodes, but sparsely inter-connected to other groups in the graph. For each community and its members, a summary is generated in a bottom-up, hierarchical manner. These summaries provide a general outlook on the data – i.e., principal entities, relations and claims in the community – and act as contextual information during the querying stage.
Query. At querying timeG,raphRAG ofers two strategies, fit to the information need: global search and local search. Considering the hierarchical nature of the community structure, queries can be answered leveraging the community summaries from diferent levels. Whether a particular hierarchy level in the community ofers the best balance of summary detail and scope for general sense-making questions or not is still an open questGiloonba.l search is suitable for holistic, comprehensive queries that require reasoning over the entire data corpus and community summarWiesh,aet.gar.,e“the top ifve themes in the data? ”. Global search implementsmaap-reduce strategy. For a given community level, the summaries are randomly shufled and divided into chunks of fixed size. mAatp step, intermediate answers are generated in parallel, and the LLM scor[e0s, 1in00a]range how relevant to the query each of them is; answers scoring 0 are excluded. Inretdhuece step, intermediate answers are ranked according to their relevance and iteratively aggregated into a new context, until the token limit i reached. The final context is employed to generate the global answer. The quality of the global search answer is afected by the level of the community hierarchy chosen for getting community reports. Lower hierarchy levels, with their detailed reports, tend to yield more thorough responses, but also increase the time and resources for generating the response due to the quantity of reports.
Local search instead is appropriate for queries reasoning on precise entities occurring in the documents, e.g. “What are the healing properties of chamomile?”. The local search technique locates a group of entities within the knowledge graph that are semantically linked to the user’s input. These entities act as gateways into the knowledge graph, facilitating the retrieval of additional pertinent informatio including associated entities, relationships, entity covariates, and community reports.
The experimental setting consists of an interface to let PA domain experts query the model about the Public Contract Code and evaluate the correctness, completeness and fluency of the3a.1n)s.wers ( Then, users filled out a satisfaction questionnaire for assessing system usability and users’ resistance to integrate this tool in their work3flo.2w).(
This experiment involves 16 expert users with heterogeneous backgrounds. In order to have a demographic overview, users were asked questions about their age, educational qualification, profession, years of experience and IT proficiency (Q.0.1 - Q.0.5).
The age of users spans from 20 to 65 years old: 18.8% users are in their twenties, 31.3% in their thirties and only 6.3% in their forties; finally, 31.3% users are in their fities, and 12.5% in their sixties. As for theireducational qualification , 12.5% users have a high-school diploma, 12.5% a Bachelor’s degree, 50% a Master’s degree, and 25% are Doctors.
As concerns usersp’rofessional role, 6.3% is a researcher, 12.5% is a freelancer, 12.5% is a university student, 31.3% is an employee, 37.5% is member of a professional order, e.g., lawyers, accountants, consultant, engineers.
Users’years of experience span from 1 to 37, with an average of 13 years. Users’ aveIrTapgreoficiency , on a 5-point Likert scale (1 - Very Low, 5 - Very High3),.4i4s.
For each answer to their query, users were asked to evalucoartrecittnsess, completeness andfluency on a 5-point Likert scale (1 - Strongly Disagree, 5 - Strongly Agree). Users were allowed to choose between local and global search (S2e.2c).. A total of 73 queries was asked. Ta1blilelustrates the questions and the mean, variance and standard deviation of their score.
Correctness achieves a mean score3.o8f0, and the highest variance and standard deviation scores. On the other hand, completeness achieves the lowest mean3s.7c4o.rTe,hese results are backed up by the answers on how to improve the system (S3e.c2.), that stress the importance of providing more specific answers and the respective article references.
Finally, as expected by an LLM, fluency obtains the highest mean sc4.o4r8e,,and the lowest variance and standard deviation scores.
According to the standard ISO 9241-11, system usability can be measured in terms of efectiveness, eficiency, and satisfaction. TheSystem Usability Score (SUS) was proposed by John Brooke in 1986 [22], and it proved to be intuitive and solid over hundreds of studies. Today, the SUS is still widely used to measure the usability of websites and applicat2i3o]n.sTh[e survey consists of 10 standard questions with the 5-point Likert scale (1 - Strongly Disagree, 5 - Strongly Agree). The SUS score for each survey participant is computed as in Equat1i,oannd assumes values in the ran[g0,e100]. and are the scores assigned to odd and even numbered questions in the questionnaire.
The proposed system achieve8s2.19, notably above the margin of the admissible range, which guidelines [24] state to be 68.
= ((∑ − 5) + (25 − ∑ )) ∗ 2.5 (1) I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
For further insights on the likelihood that users would recommend our system with, we compute the Net Promoter Score (NPS). The idea behind the NPS, proposed by Bain&7C,ois. to divide the users into promoters, passives anddetractors of the item, based on their answer: users providing ratings between 10 and 9 are considered promoters, between 8 and 7 are passives and finally, from 6 to 0 are detractors. NPS consists of a single question Q.3.H1,ow“ likely is it that you would recommend this system to a friend or colleague?”. The NPS is computed as in Equatio2nand assumes values in[−100, +100]. Any score above20 is considered encouraging, wher5e0asis excellent and abo8v0e first-rate.
The proposed system has 50% promoters and 25% detractors, therefore the obtained NPS is 25. = − (2)
To collect detailed feedback from the users about the system, the survey includes an open-ended
question Q.4.1, H“ow could we improve our tool?”. Four users answered and they all agre(ie) tinoclude
the references to the provided informa(tiio)nbe,tter detail the answers with references to the exact
articles, norms and legislative texts. To summarise, although SUS and NPS results look contrasting at
ifrst, keep in mind that the SUS questionnaire addresses the usability of the system, not how good the
system is at its intended task. High variance and standard deviation of correctness and completeness in
Table1, and answers to Q.4.1 justify the obtained NPS. The NPS score is in line with the scores obtained
in other experiments in automating PA tas4k]sa:c[hieved34.4, [
Finally, we conducted a separate study to evaluate howresmisutcahnce users would put up when integrating the system in their workflow. To this purpose, we consider the following four constructs proposed in [25, 26]: • Perceived Value (PVL). Users’ overall evaluation of the costs and benefits of adopting the tool, as determined by their attitude towards the change; • Switching Benefit (SWB). Users’ perception of the benefit that will derive from the adoption of the new tool; • Switching Cost (SWC). Users’ perception of the costs and eforts required to switch or integrate the new tool; • Self-eficacy For Change (SFC). Users’ perception of their ability to easily adapt to the new tool.
The survey consists of 14 questions with the 7-point Likert scale (1 — Strongly Disagree, 7 — Strongly Agree). Table3 illustrates the questions.
Table4 illustrates the statistics of the survey related to the users’ resistance, in terms of mean, variance and standard deviation. The three positive constructs – i.e., Perceived Value, Switching Benefit and Self-eficacy For Change – scored a mean value above 5, while the negative construct Switching Cost scored a mean value of 3. These values are indicators of a very positive users’ feedback. This insight is supported by the low variance and standard deviation values, suggesting that most users share a positive viewpoint on the system.
For a comprehensive analysis, the Pearson correlation coeficioefnthe mean scores obtained by each construct was computed. The results show a strong positive correlation between the Perceived Value and Switching Benefit, and a negative correlation between Switching Cost and every positive construct.
This paper illustrates a preliminary application of MicrGorsoaft’pshRAG on the corpus of the Italian Public Contract Code. We leverageatuhtoe prompt tuning feature to obtain prompts tailored to 7www.bain.com/insights/introducing-the-net-promoter-system-loyalty-insights/ the corpus. These adaptations made the persona and task descriptions, and examples for few-shot prompting, more domain-specific. The experimental setting consists of an interface to let PA domain experts query the model about the Public Contract Code and evaluate the correctness, completeness, and fluency of the answers. The users then filled out a satisfaction questionnaire to assess system usability and users’ resistance to integrate this tool into their workflow. Results reveal a general users’ satisfaction with the system: it achieves a S8U2S.1o9fand a NPS of25. Questions for assessing the correctness, completeness and fluency of the answers all achieve a mean score3.a7b0.oCveorrectness and completeness mean scores are backed up by the answers on how to improve the system, that highlight the significance of giving more specific answers and the respective articles references. Results of the survey on users’ resistance make clear that users consider this tool beneficial to their workflow: positive constructs obtain a mean value above 5, whereas the negative construct 3. Altogether, expert users consider the proposed system a valuable and helpful tool for their way of working.
Concerning future works, according to the answers to the open-ended question for tool improvements, it emerged that providing the textual reference to exact articles and regulations would make the system more reliable and trustworthy. Moreover, performing a comparative analysis of vector-based RAG and Graph RAG techniques may provide interesting insights. Finally, modelling legislation changes over time is a non-trivial task that would bring the proposed system a step further.
We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), Spoke 6 Symbiotic AI (CUP H97G22000210007) under the NRRP MUR program funded by the NextGenerationEU.
During the preparation of this work, the authors used Grammarly for sentence polishing, and ChatGPT for aiding the translation of legal terminology and expressions. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [17] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al., Qwen2.
5 technical report, arXiv preprint arXiv:2412.15115 (2024). [18] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023). [19] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al., Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312.11805 (2023). [20] A. Hogan, E. Blomqvist, M. Cochez, C. d’Amato, G. de Melo, C. Gutiérrez, S. Kirrane, J. E. Labra Gayo, R. Navigli, S. Neumaier, A.-C. Ngonga Ngomo, A. Polleres, S. M. Rashid, A. Rula, L. Schmelzeisen, J. F. Sequeda, S. Staab, A. Zimmermann, Knowledge Graphs, number 22 in Synthesis Lectures on Data, Semantics, and Knowledge, Springer, 2021. UhRtLt:ps://kgbook.or.g/doi:10.2200/ S01125ED1V01Y202109DSK022. [21] V. A. Traag, L. Waltman, N. J. Van Eck, From louvain to leiden: guaranteeing well-connected communities, Scientific reports 9 (2019) 1–12. [22] J. Brooke, et al., Sus-a quick and dirty usability scale, Usability evaluation in industry 189 (1996) 4–7. [23] J. R. Lewis, The system usability scale: past, present, and future, International Journal of Human–
Computer Interaction 34 (2018) 577–590. [24] J. Sauro, J. R. Lewis, Quantifying the user experience: Practical statistics for user research, Morgan
Kaufmann, 2016. [25] I. Mahmud, T. Ramayah, S. Kurnia, To use or not to use: Modelling end user grumbling as user resistance in pre-implementation stage of enterprise resource planning system, Information Systems 69 (2017) 164–179. [26] H.-W. Kim, A. Kankanhalli, Investigating user resistance to information systems implementation: A status quo bias perspective, MIS quarterly (2009) 567–582.