(ABPMSs) embody new human-centered information systems distinguished by significant flexibility, autonomy, and extensive conversational and self-enhancement abilities. [1]. Thus, Artificial Intelligence (AI) expands conventional process-aware Decision Support Systems (DSS) to facilitate prompt and effective decision-making by elucidating the underlying factors influencing the decisions [2]. Integrating ABPMSs into human workflows may introduce shifts in workforce dynamics, potentially leading to a lack of trust [3]. One possible remedy for this challenge is the incorporation of Conversational Systems (CSs). The emergence of CSs presents a promising avenue for enhancing Business Process Management (BPM) initiatives, significantly empowering ABPMSs [4,5]. The adoption of Large Language Models (LLMs) could push substantial advancements in these systems [5]. LLMs represent an emerging class of machine learning models showcasing great performance in accomplishing various Natural Language Processing (NLP) tasks [6]. Thanks to their huge advantages, practitioners are progressively utilizing LLMs across various domains, gaining significant benefits for industries and business operations while reshaping the dynamics of human interaction with management systems [7]. Notably, LLMs have been transforming several organizations towards the paradigm of autonomous enterprise and enable ABPMSs to hold a central position in assisting human activities and decisions across the system life cycle. Indeed, starting from business processes, LLMs should transcend local reasoning contexts, support the management of diverse scenarios, and enhance the business activities understanding [7]. In front of the recognized potentiality of LLMs to assist human decisions in the business landscape [1], this topic is few explored in literature [7] and, as far as we are aware, an empirical validation regarding the efficacy of LLMs for process-aware decision support is missing. In this research context, our work presents an innovative methodology for business process analysis leveraging the usage of LLMs to develop a conversational process-aware DSS. We propose to adopt a process-aware Retrieval-Augmented Generation (RAG) [8] framework to extend process-and domain-specific knowledge, in the direction of improving the conversational capability of a LLM to respond to business process-related inquiries. The overall system supports the user in a wide range of process comprehension and execution tasks using natural language. Our work evaluates the proficiency of the methodology in producing precise and contextually appropriate responses to process-related questions within different settings. In particular, we investigate the effi-cacy of the approach in a real-world scenario within the realm of public administration.
As asserted in [4], the integration of CSs holds significant potential for enhancing ABPMSs. Numerous methodologies have emerged in recent years directed at leveraging the capabilities of CSs to enhance various critical areas within BPM [5].
In the sub-field of Descriptive Process Analytics, describing current business processes and identifying problems and potential improvements, NLP and neural architectures, proved their effectiveness in extracting process models from natural language descriptions [9,10,11]. Conversely, expressing business process models in natural language aids human comprehension [12,13]. Moreover, conversational interfaces further enhance understanding and accessibility of process mining findings [14,15].
Predictive Process Analytics concerns building predictive models to forecast the future state and performance of business processes. Specifically, current trends in this area are centered around the development of conversational interfaces to assist the what-if analysis of digital process twins [16,17] and predictive process monitoring [18,19,20].
Prescriptive Process Optimization primarily focuses on improving processes, often by translating insights into actionable steps aimed at enhancing process execution. CSs designed for this BPM area mainly support automated process optimization, suggesting adjustments to optimize process performance across various indicators [21,22]. Additionally, these systems contribute to prescriptive process monitoring, providing real-time recommendations for actions to be taken, as illustrated in [23].
Augmented Process Execution embodies the concept wherein system-driven management actively oversees business process execution, with human operators providing support as needed. In this sub-field, various conversational agents have been developed to facilitate seamless interaction between systems and human users [24,25,26]. Furthermore, Robotic Process Automation (RPA), which involves creating software robots to automate repetitive tasks on application user interfaces, will likely benefit from the combination with CSs. Such integration enables the automation of business processes [27,28,29], and aids in identifying suitable routines for automation through natural language interaction [30,31].
In this study, we present a business process-oriented LLM framework, better detailed in [32]. The steps utilized for answering queries pertaining to business processes are summarized in Figure 1. The overall architecture comprises two major phases: Knowledge Augmentation and Querying.
The process-aware LLM pipeline starts by considering a business process model in input, resulting in the production of multiple chunks. This operation is undertaken to facilitate the LLM's understanding in generating responses. In this study, we utilized a Directly-Follows Graph (DFG) representation expressed in natural language.
In fact, chunking aims to partition broad textual content into more manageable segments, enabling the LLM to ingest only relevant context and overcoming limitations imposed by its context window. To ensure meaningful chunks and mitigate unnatural segmentation of the process model, two distinct chunking strategies were evaluated: fixed-size and recursive chunking.
Subsequently, the framework proceeds to transform the raw input chunks into model embeddings for storage in a vector index. These embeddings are dense, lowdimensional vectors designed to encapsulate semantic information and contextual relationships necessary for the successive retrieval and generation operations.
Afterward, the business process model embeddings are stored within a specialized vector database to enable efficient retrieval. This retrieval procedure is enacted through semantic search that, in our case, relies on cosine similarity.
The Querying stage begins with the retrieval of the pertinent process model chunks needed for the crafting of precise responses to the process-related questions. In particular, this retrieval step involves fetching relevant process chunks from the vector store through semantic search utilizing cosine similarity. Following this, these segments, along with the user question, are fed into an LLM to generate an answer.
Ultimately, to offer contextually grounded answers based on the user query and the retrieved information, the proposed framework relies on two primary components: a LLM and its associated tokenizer. Initially, a prompt is formulated by merging the user query with the previously retrieved process context. Subsequently, the tokenizer converts the prompt into a format comprehensible by the model. Eventually, the prompt is fed to the LLM to generate contextually relevant answers. In particular, our process-aware approach integrates the Llama 2 13B [33] model as the LLM.
We performed a preliminary validation on the adoption of the proposed framework by applying it to a real public administration procedure. The process model, illustrated in Figure 2, involves the reimbursement of expenses for missions, a critical procedure within a university. This administrative process entails the processing of expense reports submitted by employees and the subsequent decision to either reimburse or reject these reports. In particular, the process was analyzed using textual DFG descriptions of activities and sequence flows.
The proposed framework, being rooted in generative models, provides feedback to users in natural language. To assess its effectiveness in aiding users' comprehension of business processes, the validation encompasses assessing the accuracy of the answers concerning the entities and relationships present in both the process model and the response of the LLM. The conclusion derived from this research effort centers on evaluating the approach's overall effectiveness in assisting business process users and discussing its potential applications in real-world scenarios.
All the evaluations are performed using the reimbursement process model previously introduced, represented using the DFG expressed in natural language.
The queries adopted in this evaluation require, to be answered, to recognize both structural and behavioral information within the model. By structural information, it is considered the presence of activities, events, and gateways in the process model whereas behavioral information encompasses details concerning the sequence flows linking these entities.
Specifically, for structural information correctness analysis, we queried the presence of specific activities within the business process model, prompting the pipeline to answer with a simple "yes" or "no" and to provide relevant contextual references if available.
When assessing behavioral features, inquiries were expressed to check the presence of sequence flows between specified activities in the process representation. The LLM was prompted to state their existence in a binary manner, reporting contextual references.
Striving to obtain a thorough evaluation, we analyzed all single-pass transitions, an equivalent number of sequence flows between activities present in the model but not directly connected, and the same number of flows linking tasks that do not belong to the process.
First, we assessed the performance of the RAG-based framework in comparison to the basic version of the language model for responding to the queries within the context of the reimbursement process model. Specifically, we estimated the capability of the LLaMA 2 13B model and the RAG-based pipeline in addressing related to business processes, employing accuracy as the measure.
For this reason, we designed an evaluation approach for assessing the performance of the framework relying on binary response questions (expecting either a "yes" or a "no" as allowed answers) to allow a rigorous assessment of the provided answers. The accuracy quantifies the proportion of exact predictions generated by the LLM in answering the user's questions out of the total responses provided. We classify predictions given by the framework as true positives (TP) when they correspond to positive expected outcomes and as true negatives (TN) when they match negative expected outcomes. Vice versa, false positives (FP) arise when the approach produces positive answers opposite to negative expectations, whereas false negatives (FN) derive from negative answers generated by the framework despite positive expected ones.
Subsequently, we estimated the effects of employing various chunking techniques within the process-aware LLM pipeline, alongside investigating how prompt engineering can further augment the framework's performance. Fixed-size and recursive chunking with different sizes are tested.
In both cases, the accuracy (reported in Formula 1) of the framework in answering the queries is evaluated.
We carried out this evaluation employing an oracle that considers both the query and the corresponding binary response as input. Such oracle compares the answers of the pipeline with the expected ones and computes the accuracy as the ratio of correct predictions to the total number of tests conducted in that particular assessment.
In our experimentation, we found that by retrieving the top 20 chunks, we were always able to capture a comprehensive overview of the process model, enabling the language model to generate grounded responses. The experiments were conducted on a workstation running the Linux/Ubuntu 22.04.3 LTS operating system and equipped with an NVIDIA A100 GPU.
We proceed to analyze the results obtained during the evaluation phase under various experimental conditions.
The results in terms of accuracy for the basic LLM and the RAG-based pipeline on the reimbursement DFG model described in natural language are presented in Table 1.
The table demonstrates a notable improvement in accuracy upon utilizing the RAG-based LLM, which is consistent with our expectations for the test. This enhancement exhibits an acceptable performance level (72.37 percent) for the framework, relying on the natural language representation to drive more informed and accurate decisionmaking.
Our observations revealed instances of hallucination, wherein the pure LLM would provide responses despite lacking pertinent information about the process model, occasionally asserting familiarity with certain activities even when such knowledge was absent.
Table 2 illustrates the accuracy computed using various chunking methods, including no chunking, fixed-size chunking, and recursive chunking.
Comparable outcomes are achieved through the usage of a fixed-size strategy and a recursive technique for chunking leveraging the natural language representation. In both cases, the ideal size for the chunks is identified as 128 tokens with a 10-token overlap. We can attribute this observation to the relatively modest scale of the process model, which causes its content to be nearly encapsulated within a single chunk. Additionally, the above consideration clarifies why the absence of chunking yields analogous results.
In conclusion, this work introduced a business processaware LLM, an innovative framework designed to facilitate actionable conversations and support process-aware DSSs, thereby laying the ground for intelligent interaction with ABPMSs. The proposed methodology, tailored for aiding business process analysis, aims to enhance the conversational skills of LLMs in the business process context. This objective is realized through the development of a RAG-based architecture, which extends its knowledge of the structural and behavioral aspects of process models by ingesting contextual information concerning specific inquiries. Consequently, the processaware framework is equipped to assist users in understanding and executing business processes through a natural language interface. Additionally, we assessed the performance of the process-aware LLM in providing precise and pertinent answers to the queries posed by the users across diverse evaluation scenarios.
In future research within the domain of process discovery [34], we intend to delve into the analysis of the business process execution information and explore the impact of different embedding models on the developed technique. Furthermore, investigating the integration of the framework with symbolic AI solvers to embed reasoning capabilities could present another intriguing avenue for future work.
The work of Angelo Casciani has been carried out in the range of the Italian National Doctorate on AI run by Sapienza.