Conversational agents provide new modalities to access and interact with services and applications. Recently, they saw a backfire in their popularity, due to the recent advancements in language models. Such agents have been adopted in various ifelds such as healthcare and education, yet they received little attention in public administration. We describe as a practical use case a service of the portal that provides citizens of the Italian region of Friuli-Venezia Giulia with services related to their own Electronic Health Records. The service considered allows them to search for the available doctors and pediatricians in the region's municipalities. We rely on the use case described to propose a model for a conversational agent-based access modality. The model proposed allows us to lay the foundation for more advanced chatbot-like implementations which will use also alternative input modalities, such as voice-based communication.
They are well-known for some time; indeed, their first
mention can be dated back to 1966, with the ELIZA
system [
far. Even though ChatGPT is not free from criticism, in
particular about the quality of the generated content [
sion of its own Bing search engine.1 OpenAI published a LGOBE
0000-0002-7337-7592 (M. Soprano); 0000-0002-9191-3280 (K. Roitero); 0000-0002-0144-3802 (V. D. Mea); 0000-0002-2852-168X (S. Mizzaro) CEUR htp:/ceur-ws.org ISN1613-073 https://news.microsoft.com/the-new-Bing/ technical report about GPT-4 [6] in March 2023. Such a model is, indeed, the latest iteration of the GPT family.
Turning back to conversational agents in general, they can have a great impact on society; they provide new modalities to interact with various services and may help the unexpected popularity of chatbot-like conversational agents could be beneficial also to less explored fields, such as public administration. The available implementations are mostly designed to ease the communication between administrative bodies and people, by allowing the latter to obtain answers for their issues without the involvement of staf [ 7]. For instance, Anastasiou et al. [8] developed a multilingual chatbot service to provide expatriates who enter a new country some help in dealing with procedures such as application for residence.
can ofer in a digital fashion, one of particular interest is providing citizens with access to their own Electronic Health Record (EHR). The EHR represents and collects demographic, administrative, and clinical patient-centred data [9]. Access by citizens to their own EHRs is becoming an integral part of healthcare systems worldwide. For instance, the Italian Government published, back in 2014, the guidelines for the presentation of regional projects plans for the creation of the EHR, known as “Fascicolo Sanitario Elettronico”.2 As of today, every Italian region provides some kind of portal to allow citizens to access their EHRs, and the government’s agency for the digi
Attribution 4.0 International (CC BY 4.0).
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License tal innovation provided in February 2023 the technical of today, they have been studied and adopted within var- the elderly, disabled, visually-impaired people, etc. Also, specifications for interoperability between the regional in the case of ChatGPT, other aspects such as the data systems. We believe that relying on a conversational focus or the training procedure are more important [16]. agent-based access modality for citizens to their own EHRs systems can assist providers in delivering high EHRs is an important matter to address. quality care to patients. Indeed, Kruse et al. [17] assessed
Let us now focus again on conversational agents. in 2018 the validity of EHRs to improve the quality and Given that a public administration portal is used by peo- eficiency of healthcare. Furthermore, Menachemi and ple of varying ages, digital skills, etc., building an efec- H. Collum [18] describe in detail in their recent work tive conversation model is not an easy task. A way to why society needs such systems along with their advancope with such dificulty is relying on all the social cues tages and clinical outcomes. Tapuria et al. [19] reviewed typical of human conversation such as small talk, gen- 74 papers about providing patient access to their own der, age, gestures, facial expressions to which humans electronic health records. They found out that the majorreact, and many others. The agent should thus display ity of papers (54 out of 70) showed positive outcome or and employ such cues whenever possible. Feine et al. benefits by accessing to their EHRs via patient portals, [10] provide a taxonomy for such social cues in conver- and de Mello et al. [20] propose a taxonomy for semantic sational agents, finding 48 of them, and Amatulli et al. interoperability in EHRs. [11] show that they have an impact on the tendency of Turning to conversational agents, their usage can rely older consumers’ choice of contemporary over traditional on natural language vocal cues. The Conversational Inproducts. We also hypothesize that the usage of an ad- formation Seeking [21] (CIS) research area involves invanced ChatGPT-like language model to improve part of teraction sequences between one or more users and an the interaction (if not the whole) of the citizen with the information systems where the possible interactions are EHR portal could be interesting to implement and study. primarily based on natural language dialogue, while other types of interaction can still be included. Understanding the characteristics of people that use conversational 2. Aims agents can be useful to improve an existing implementation. Gkinko and Elbanna [22] propose a taxonomy to describe chatbot users according to four diferent types: early quitters, pragmatics, progressives and persistents.
Furthermore, Parmar et al. [23] show that several healthfocuses apps that use chatbots exist and they are often used to address gaps in healthcare quality.
In this paper, we propose a conversational agent-based access modality for citizens to their own EHR in the form of a conversation model for the implementation of a first working prototype. Such a model allows interaction with the relevant databases and APIs of the public administration services in a seamless and eficient manner, providing users with a more user-friendly and streamlined experience by allowing them to input text in a natural and conversational manner, rather than requiring them to follow a rigid set of rules.
The model we propose is focused on implementing a specific and simple use case of the portal implemented by the Italian region Friuli-Venezia Giulia to provide EHR access to their citizens, known as SeSaMo,3 whose name is derived from the Italian translation of “mobile health services”.
The SeSaMo portal provides a set of so-called “quick services” (English translation of “servizi fast”), which complement the EHR access and browsing features and do not require explicit user authentication. Such services allow citizens, for example, to search for doctors and pediatricians, surgeries and pharmacies. Let us consider the first case. A citizen of the Italian Friuli-Venezia Giulia region may search for a doctor or a pediatrician by using the corresponding quick service.4 Such a service 3. Related Work is of critical importance for Italian citizens because, for instance, each of them must choose the general practiChatGPT is a sibling model of InstructGPT [12], which tioner among the doctors available in the municipality is trained to follow instructions in a prompt and provide of residence and they is the one a citizen should contact a detailed response. They are derived from GPT-3 and ifrst to address any health-related problem. The service, then optimized using human feedback in the form of mid- thus, allow citizens to find every detail needed about training reinforcement learning [13]. When building a their general practitioner of choice, including the workgeneral language model for various types of tasks its size ing hours and alternative professionals to contact if they seems to matter [14], as happened for GPT-3 over its pre- is not available. Figure 1 summarizes the service’s usage decessor GPT-2 and other models [15]. However, when by providing a high-level use case diagram. We use it in restricting the focus to a narrower scope, like chatbots 3https://sesamo.sanita.fvg.it/ 4https://sesamo.sanita.fvg.it/sesamo/#/il_mio_medico Figure 3 proposes a graph, as a conversation model between a citizen who is using the quick service of the SeSaMo portal described in Section 4 and the implementation of a chatbot-like conversational agent. Each color-coded block of the graph corresponds to a message prompted to the user by the conversational agent. Each color has a diferent interpretation. A white block implies that a message is simply shown to the user, while a yellow one means that they is shown a button-based choice to be performed to proceed. A green block means that the user is required to provide a free textual answer.
A red block in the graph implies that the chatbot performs internal business logic, whose outputs are used in the following messages. On the other hand, a violet block indicates a conditional logic step that can branch the conversation like when a yellow block is shown, but without prompting any message to the user. Lastly, a blue block means that the user is prompted with multiple looped messages. The messages which are shown within each block aim to use the textual social cues described by Feine et al. [10, Table A1].
The conversation works as follows. Initially, the user is asked whether they knows the name of a given doctor.
If that is the case, they is required to provide it; otherwise, the agent proposes to filter the available doctors by municipality. This can be useful if, indeed, the user is not aware of the doctors available in the municipal- The model shown in Figure 3 does not consider more ity of residence. Then, the agent fetches and filters the advanced interfaces or input methodologies, such as list of available doctors. If none of them is found, it ac- voice-based communication with the agent. This is due to knowledges the user by asking whether they wants to the fact that the initial prototype of the chatbot-like constart searching from scratch. On the other hand, if the versational agent is going to be implemented to support agent finds at least one doctor the conversation contin- the traditional text-guided conversation loop described ues. If a single doctor is found, the details are shown to by the graph. The contribution that a ChatGPT-like lanthe user; otherwise, the agent checks if there are both guage model could provide to improve the conversation doctors and pediatricians in the resulting set. In such a will be considered in future prototypes and for, most case, the user is asked whether they is interested in the likely, more advanced and complex use cases related to former category or the latter. Then, the agent filters the EHR access, browsing and usage. However, we can alresult set and checks once again the number of remaining ready argue that ChatGPT-like models will likely allow us, doctors. If there is a single one of them only, the conver- for instance, to implement the green and yellow blocks sation goes on and the details are shown; otherwise, the of the graph by letting the users write messages using agent prompts an additional message and asks the user natural language only. to explicitly select the desired doctor. Another aspect that must be considered to address
To summarize, the conversation can reach the point the evolution of the conversation in the future is the uswhere the details of a single doctor are shown by follow- age of alternative input modalities, such as voice-based ing the three possible branches described. Then, showing communication provided by devices such as Amazon the doctor’s details involves reporting the information Alexa [24]. Indeed, voice-based devices are ubiquitously about each surgery of the one chosen, including its ad- available and there are studies concerning the design dress, working hours and phone number, as happens in of efective interfaces [ 25]. Even though there are secuthe portal’s interface shown in Figure 2. As a last step, rity concerns that need to be mitigated [26], such voicethe agent checks if a list of doctors associated with the based devices have demonstrated their usefulness, for selected one exists. The agent thus asks explicitly the instance, by helping prevent and manage chronic and user whether they is interested in browsing them and mental health conditions [27]. We thus believe that using if they agrees the conversation loops, as can be seen in voice-based interfaces can fruitfully improve the quality the bottom right part of Figure 3. If the user refuses, the and efectiveness of our conversational agent proposal conversation can end and they is allowed to write some for EHR access and browsing in public administration. kind of textual feedback or comment.
Technologies that allow implementing a chatbot-like con- prototyping a chatbot-like conversational agent to allow versational agent already exist. Several social media plat- citizens of the Friuli-Venezia Giulia Italian region to acforms such as Telegram, Slack and others ofer native cess the quick service of the SeSaMo portal and search APIs and interfaces to build such agents, given that they for doctors. The considered use case allows us to lay will be available only for the users of such a platform. the foundations for future versions which will provide To address such a limitation, integrated development more advanced EHR-related features. The recent and environments for building conversational agents exist, popular language models could be used to enhance fusuch as the Microsoft Azure Bot Services. 5 Indeed, one ture prototypes, for instance by asking the users to state of the most interesting features is that it allows deploy- their goal, into a set of actions that the conversational ing easily the developed agents on multiple platforms agent can understand using natural language processing such as Telegram, Slack, and more. Another aspect to techniques. Such models can also capture the nuances of consider is how to evaluate the performances of the con- natural language, including paraphrasing and ambiguity; versational agent. Kohavi and Longbotham [28] describe this will allow us to handle a wider range of user inputs several approaches to evaluate prototypes using online and provide more accurate responses. controlled experiments. The most simple experimental Lastly, it will be essential in the future to ensure that setup is the A/B testing procedure; the default version of the conversational agent adheres to ethical principles, a system is evaluated against its updated or new version. such as transparency, fairness, and privacy. This can be We thus argue that the initial prototype that implements achieved through various techniques, such as providing the model shown in Figure 3 could be evaluated in such clear explanations of how the agent works, ensuring that a fashion against a subsequent iteration of the conver- it is not biased against certain groups, and protecting sational agents that uses to some extent a ChatGPT-like user data from unauthorized access or misuse. model.
Evaluating the performances of our conversational Acknowledgments agents will require efectiveness metric which should consider multiple factors such as engagement, conver- This work has been partially supported by the “Departsation length, and others. Irvine et al. [29] define, in ment Strategic Plan of the University of Udine - Intertheir recent work, four intuitive evaluation metrics as departmental Project on Digital Governance and Public proxies to measure the level of engagement of conversa- Administration” (Jan. 2021 – Dec. 2025). We thank Paolo tional agents. The mean conversation length measures Coppola, who provided helpful comments and advice. the average number of user queries over multiple conversation sessions, defined as ordered sets of user and agent response pairs. This metric could be particularly References useful if we decide, for instance, to track the previous conversations of each user. A common functionality of conversational agents is allowing the user to regenerate another response to their message. The graph proposed in Figure 3 is rather simple, yet the loop shown in the left part of the graph, triggered when the agent does not retrieve any doctor, could be seen in such a way. When the conversation with the user ends with the green block shown in the right part of the graph, the agent could ask the user to provide feedback by rating their answers using a given rating scale and thus measure user satisfaction. Lastly, the user retention of the conversational agent could be computed to evaluate their engagement after the first conversation; however, we believe that such a metric is less important for evaluating our agent, given that we aim to target a public administration service and not a commercial platform and we are not interested in its monetization or similar aspects.