AI-Augmented Business Process Management Systems (ABPMSs) are innovative information systems with increased flexibility, autonomy, and conversational capability. These systems can be boosted by Large Language Models (LLMs), renowned for their ability to handle natural language processing tasks. Nevertheless, no significant empirical validations exist about their usefulness in process-driven decision support. In this study, we propose a business process-oriented LLM framework, for enacting actionable conversations with workers involved in a business process, leveraging Retrieval-Augmented Generation (RAG) to enrich process-specific knowledge. The methodology has been assessed to evaluate its capacity to produce precise responses to inquiries posed by users within a public administration context. The preliminary study shows the framework's ability to identify specific activities and sequence flows within the targeted process model, thereby providing valuable insights into its potential for improving ABPMSs.
to their huge advantages, practitioners are progressively
AI-Augmented Business Process Management Systems utilizing LLMs across various domains, gaining
signifi(ABPMSs) embody new human-centered information sys- cant benefits for industries and business operations while
tems distinguished by significant flexibility, autonomy, reshaping the dynamics of human interaction with
manand extensive conversational and self-enhancement abil- agement systems [
In this study, we present a business process-oriented LLM framework, better detailed in [32]. The steps utilized for answering queries pertaining to business processes are summarized in Figure 1. The overall architecture comprises two major phases: Knowledge Augmentation and Querying. cacy of the approach in a real-world scenario within the realm of public administration.
potential for enhancing ABPMSs. Numerous
methodologies have emerged in recent years directed at leveraging
the capabilities of CSs to enhance various critical areas Knowledge Augmentation The process-aware LLM
within BPM [
In the sub-field of Descriptive Process Analytics, describ- in input, resulting in the production of multiple chunks.
ing current business processes and identifying problems This operation is undertaken to facilitate the LLM’s
unand potential improvements, NLP and neural architec- derstanding in generating responses. In this study, we
tures, proved their efectiveness in extracting process utilized a Directly-Follows Graph (DFG) representation
models from natural language descriptions [
Conversely, expressing business process models in natu- In fact, chunking aims to partition broad textual
conral language aids human comprehension [
Predictive Process Analytics concerns building predic- the process model, two distinct chunking strategies were tive models to forecast the future state and performance evaluated: fixed-size and recursive chunking. of business processes. Specifically, current trends in this Subsequently, the framework proceeds to transform area are centered around the development of conversa- the raw input chunks into model embeddings for storage tional interfaces to assist the what-if analysis of digital in a vector index. These embeddings are dense, lowprocess twins [16, 17] and predictive process monitoring dimensional vectors designed to encapsulate semantic [18, 19, 20]. information and contextual relationships necessary for
Prescriptive Process Optimization primarily focuses on the successive retrieval and generation operations. improving processes, often by translating insights into ac- Afterward, the business process model embeddings tionable steps aimed at enhancing process execution. CSs are stored within a specialized vector database to enable designed for this BPM area mainly support automated eficient retrieval. This retrieval procedure is enacted process optimization, suggesting adjustments to optimize through semantic search that, in our case, relies on cosine process performance across various indicators [21, 22]. similarity.
Additionally, these systems contribute to prescriptive process monitoring, providing real-time recommendations for actions to be taken, as illustrated in [23].
Augmented Process Execution embodies the concept wherein system-driven management actively oversees business process execution, with human operators providing support as needed. In this sub-field, various conversational agents have been developed to facilitate seamless interaction between systems and human users [24, 25, 26]. Furthermore, Robotic Process Automation (RPA), which involves creating software robots to automate repetitive tasks on application user interfaces, will likely benefit from the combination with CSs. Such integration enables the automation of business processes [27, 28, 29], and aids in identifying suitable routines for automation through natural language interaction [30, 31].
Querying The Querying stage begins with the retrieval of the pertinent process model chunks needed for the crafting of precise responses to the process-related questions. In particular, this retrieval step involves fetching relevant process chunks from the vector store through semantic search utilizing cosine similarity. Following this, these segments, along with the user question, are fed into an LLM to generate an answer.
Ultimately, to ofer contextually grounded answers based on the user query and the retrieved information, the proposed framework relies on two primary components: a LLM and its associated tokenizer. Initially, a prompt is formulated by merging the user query with the previously retrieved process context. Subsequently, the tokenizer converts the prompt into a format comprehensible by the model. Eventually, the prompt is fed to the LLM to generate contextually relevant answers.
In particular, our process-aware approach integrates the Llama 2 13B [33] model as the LLM.
using the DFG expressed in natural language.
The queries adopted in this evaluation require, to be We performed a preliminary validation on the adoption answered, to recognize both structural and behavioral of the proposed framework by applying it to a real public information within the model. By structural information, administration procedure. The process model, illustrated it is considered the presence of activities, events, and in Figure 2, involves the reimbursement of expenses for gateways in the process model whereas behavioral inmissions, a critical procedure within a university. This formation encompasses details concerning the sequence administrative process entails the processing of expense lfows linking these entities. reports submitted by employees and the subsequent de- Specifically, for structural information correctness cision to either reimburse or reject these reports. In par- analysis, we queried the presence of specific activiticular, the process was analyzed using textual DFG de- ties within the business process model, prompting the scriptions of activities and sequence flows. pipeline to answer with a simple "yes" or "no" and to
The proposed framework, being rooted in generative provide relevant contextual references if available. models, provides feedback to users in natural language. When assessing behavioral features, inquiries were exTo assess its efectiveness in aiding users’ comprehension pressed to check the presence of sequence flows between of business processes, the validation encompasses assess- specified activities in the process representation. The ing the accuracy of the answers concerning the entities LLM was prompted to state their existence in a binary and relationships present in both the process model and manner, reporting contextual references. the response of the LLM. The conclusion derived from Striving to obtain a thorough evaluation, we analyzed this research efort centers on evaluating the approach’s all single-pass transitions, an equivalent number of seoverall efectiveness in assisting business process users quence flows between activities present in the model but and discussing its potential applications in real-world not directly connected, and the same number of flows scenarios. linking tasks that do not belong to the process. First, we assessed the performance of the RAG-based 4.1. Evaluation Setting framework in comparison to the basic version of the language model for responding to the queries within the All the evaluations are performed using the reimburse- context of the reimbursement process model. ment process model previously introduced, represented Specifically, we estimated the capability of the LLaMA
related to business processes, employing accuracy as the measure.
For this reason, we designed an evaluation approach for assessing the performance of the framework relying on binary response questions (expecting either a "yes" or a "no" as allowed answers) to allow a rigorous assessment of the provided answers. The accuracy quantifies the proportion of exact predictions generated by the LLM in answering the user’s questions out of the total responses provided. We classify predictions given by the framework as true positives (TP) when they correspond to positive expected outcomes and as true negatives (TN) when they match negative expected outcomes. Vice versa, false positives (FP) arise when the approach produces positive answers opposite to negative expectations, whereas false negatives (FN) derive from negative answers generated by the framework despite positive expected ones. 4.2. Evaluation results
evaluation phase under various experimental conditions.
The results in terms of accuracy for the basic LLM + and the RAG-based pipeline on the reimbursement DFG = (1) model described in natural language are presented in + + + Table 1.
The table demonstrates a notable improvement in accu
Subsequently, we estimated the efects of employing racy upon utilizing the RAG-based LLM, which is consisvarious chunking techniques within the process-aware tent with our expectations for the test. This enhancement LLM pipeline, alongside investigating how prompt en- exhibits an acceptable performance level (72.37 percent) gineering can further augment the framework’s perfor- for the framework, relying on the natural language repremance. Fixed-size and recursive chunking with diferent sentation to drive more informed and accurate decisionsizes are tested. making.
In both cases, the accuracy (reported in Formula 1) of Our observations revealed instances of hallucination, the framework in answering the queries is evaluated. wherein the pure LLM would provide responses despite
We carried out this evaluation employing an oracle that lacking pertinent information about the process model, considers both the query and the corresponding binary occasionally asserting familiarity with certain activities response as input. Such oracle compares the answers of even when such knowledge was absent. the pipeline with the expected ones and computes the Table 2 illustrates the accuracy computed using various accuracy as the ratio of correct predictions to the total chunking methods, including no chunking, fixed-size number of tests conducted in that particular assessment. chunking, and recursive chunking.
In our experimentation, we found that by retrieving Comparable outcomes are achieved through the usthe top 20 chunks, we were always able to capture a age of a fixed-size strategy and a recursive technique for chunking leveraging the natural language representation. In both cases, the ideal size for the chunks is identified as 128 tokens with a 10-token overlap. We can attribute this observation to the relatively modest scale of the process model, which causes its content to be nearly encapsulated within a single chunk. Additionally, the above consideration clarifies why the absence of chunking yields analogous results.
aware LLM, an innovative framework designed to facilitate actionable conversations and support process-aware DSSs, thereby laying the ground for intelligent interaction with ABPMSs. The proposed methodology, tailored for aiding business process analysis, aims to enhance the conversational skills of LLMs in the business process context. This objective is realized through the development of a RAG-based architecture, which extends its knowledge of the structural and behavioral aspects of process models by ingesting contextual information concerning specific inquiries. Consequently, the processaware framework is equipped to assist users in understanding and executing business processes through a natural language interface. Additionally, we assessed the performance of the process-aware LLM in providing precise and pertinent answers to the queries posed by the users across diverse evaluation scenarios.
In future research within the domain of process discovery [34], we intend to delve into the analysis of the business process execution information and explore the impact of diferent embedding models on the developed technique. Furthermore, investigating the integration of the framework with symbolic AI solvers to embed reasoning capabilities could present another intriguing avenue for future work.
Integrating Learning, Optimization and Reasoning wards Hybrid Automation by Bootstrapping ConWorkshop (2020). versational Interfaces for IT Operation Tasks, in: [14] L. Barbieri, E. Madeira, K. Stroeh, W. van der Aalst, AAAI, 2023, pp. 15654–15660.
A natural language querying interface for process [27] P. D. Hung, D. T. Trang, T. Khai, Integrating Chatmining, Journal of Intelligent Information Systems bot and RPA into Enterprise Applications Based on 61 (2023) 113 – 142. Open, Flexible and Extensible Platforms, in: Int. [15] H. Yeo, E. Khorasani, V. Sheinin, I. Manotas, N. P. A. Conf. on Cooperative Design, Visualization and EnVo, O. Popescu, P. Zerfos, Natural Language Inter- gineering, 2021, pp. 183–194. face for Process Mining Queries in Healthcare, in: [28] G. Dan, D. Claudiu, F. Alexandra, et al., MultiProceedings - 2022 IEEE Int. Conf. on Big Data, Big Channel Chatbot and Robotic Process Automation, Data 2022, Institute of Electrical and Electronics in: IEEE Int. Conf. on Automation, Quality and Engineers Inc., 2022, p. 4443 – 4452. Testing, Robotics, 2022, pp. 1–6. [16] D. Barón-Espitia, M. Dumas, O. González-Rojas, [29] Y. Rizk, V. Isahagian, S. Boag, et al., A ConverCoral: Conversational What-If Process Analysis, in: sational Digital Assistant for Intelligent Process ICPM Demo, 2022, pp. 118–122. Automation, BPM Forum (2020). [17] M. Li, R. Wang, X. Zhou, Z. Zhu, Y. Wen, R. Tan, [30] H. van der Aa, H. Leopold, Automatically identiChatTwin: Toward Automated Digital Twin Gen- fying process automation candidates using natural eration for Data Center via Large Language Mod- language processing, Blockchain and Robotic Proels, in: Proceedings of the 10th ACM Int. Conf. cess Automation (2022) 77 – 86. on Systems for Energy-Eficient Buildings, Cities, [31] Z. Zeng, W. Watson, N. Cho, et al., FlowMind: and Transportation, BuildSys ’23, Association for Automatic Workflow Generation with LLMs, in: Computing Machinery, 2023, p. 208–211. ACM Int. Conf. on AI in Finance, 2023, pp. 73–81. [18] K. Brennig, K. Benkert, B. Löhr, O. Müller, Text- [32] A. Casciani, M. L. Bernardi, M. Cimitile, A. MarAware Predictive Process Monitoring of Knowledge- rella, Conversational systems for ai-augmented Intensive Processes: Does Control Flow Matter?, business process management, PREPRINT (Version in: Int. Conf. on Business Process Management, 1) available at Research Square (2024). doi:https: Springer, 2023, pp. 440–452. //doi.org/10.21203/rs.3.rs-4125790/v1. [19] L. Cabrera, S. Weinzierl, S. Zilker, M. Matzner, Text- [33] H. Touvron, T. Lavril, G. Izacard, X. Martinet, et al., Aware Predictive Process Monitoring with Contex- LLaMA: Open and Eficient Foundation Language tualized Word Embeddings, in: BPM Workshops, Models, 2023. arXiv:2302.13971. volume 460 LNBIP, 2022, p. 303 – 314. [34] M. L. Bernardi, M. Cimitile, C. Di Francescomarino, [20] C. Warmuth, H. Leopold, On the Potential of Tex- F. M. Maggi, Using discriminative rule mining tual Data for Explainable Predictive Process Mon- to discover declarative process models with nonitoring, ICPM Workshops 468 LNBIP (2023) 190 – atomic activities, Lecture Notes in Computer Sci202. ence (including subseries Lecture Notes in Arti[21] S. Badini, S. Regondi, E. Frontoni, R. Pugliese, As- ifcial Intelligence and Lecture Notes in Bioinforsessing the capabilities of ChatGPT to improve addi- matics) 8620 LNCS (2014) 281 – 295. doi:10.1007/ tive manufacturing troubleshooting, Advanced In- 978-3-319-09870-8_21.
dustrial and Engineering Polymer Research (2023). [22] A. Mustansir, K. Shahzad, M. K. Malik, Towards automatic business process redesign: an NLP based approach to extract redesign suggestions, Automated Software Engineering 29 (2022). [23] S. Zeltyn, S. Shlomov, et al., Prescriptive Process
Monitoring in Intelligent Process Automation with Chatbot Orchestration, in: PMAI@IJCAI, 2022, pp.
49–60. [24] T. Chakraborti, S. Agarwal, Y. Khazaeni, et al.,
D3BA: A Tool for Optimizing Business Processes Using Non-deterministic Planning, in: BPM Workshops, 2020, pp. 181–193. [25] L. F. Lins, G. Melo, T. Oliveira, et al., PACAs:
Process-Aware Conversational Agents, in: BPM
Workshops, 2021, pp. 312–318. [26] J. Bandlamudi, K. Mukherjee, P. Agarwal, et al., To