Recently, the BPM community has started experimenting with LLMs to extract process information and to generate process models from natural text with approaches such as [1,2,3,4,5]. These works indicate promising capabilities of LLMs for this task.
With ProMoAI [6,7], there is also an online tool 1 available in the literature. The tool not only allows the initial generation of a process model from text. It also offers a feedback loop for refining the generated model. ProMoAI indicates a high potential of LLMs for this task. However, the applied prompting strategy and intermediate formats impose significantly high usage fees. When we first tried the tool, we generated a 3-step process with two iterations of the feedback loop. This small experiment costed 0.8 USD in OpenAI API fees using GPT-4. While the current GPT-4o model is more cost-effective, we argue that such systems should be optimized to efficiently use LLM resources to reduce costs while maintaining high-quality output. We propose that an optimized system can potentially democratize process modeling or at least make process modeling accessible to a broader audience. We have, therefore, developed our own approach in [8] that focuses on the modeling costs in terms of the required number of tokens.
The BPMN-Chatbot implements our efficient approach for LLM-based process modeling in [8]. It is publicly available on the tool's homepage 2 , which also includes a video demonstration and further resources. julius.koepke@aau.at (J. Köpke); aya.safan@aau.at (A. Safan) 0000-0002-6678-5731 (J. Köpke)
The BPMN-Chatbot mimics the look and feel of a classical messenger application. However, the output is rendered in the form of BPMN process diagrams. A screenshot of the tool is shown in Fig. 1. First, the user provides a process description using text or voice input (see Fig. 1 [A]). The tool then generates a business process and shows it graphically. In an optional feedback loop, the user can provide feedback on the model and obtain an updated version. During the feedback loop, the user can navigate between different model versions using arrow symbols to provide feedback on a specific version (see Fig. 1 [B]).
Finally, the generated model can be downloaded as a BPMN-XML File (see Fig. 1 [C]). For evaluation purposes, the tool also includes a survey component for eliciting user feedback after the modeling session.
Example Fig. 1 shows a screenshot of the tool after one iteration of the feedback loop. In particular, the user first asked the BPMN-Chatbot to create a process model for the following scenario: When an order is received, we first check it, then we collect the items, and finally, we send the package to the customer. The system then generated a purely sequential model. In the next iteration, the user asked the tool to refine the model with the comment: If the items are not on stock, we reject the order. The resulting process shows the correct usage of a gateway to address the comment.
[A]
[B]
[C]
The tool is implemented in the form of a React single-page web application. An architectural sketch is shown in Fig. 2. We focus here on the core components Prompt Generation and Model2Model Translation.
The prompt generation component creates messages that are sent to the LLM API (currently OpenAI). The Model2Model Translation component is used to transform an intermediate process format returned by the LLM to standard BPMN that is then visualized via a rendering component that uses the bpmn.js3 library.
This is the core component of the tool, which is responsible for the low numbers of required tokens. To keep the number of tokens low, we aim for prompts with minimal overhead on top of the user input. To optimize the feedback loop costs, we only send the feedback comment and the referenced process model to the LLM. This approach is further enhanced by using an efficient intermediate JSON representation [8] for processes returned from (and sent to) the LLM. It covers the core set of BPMN elements (also used by other approaches [6,7]), providing block-structured process models We have iteratively optimized our prompt in preliminary experiments on process descriptions disjoint from our evaluation data sets. This process has led to the following prompt shown in Listing 1.
You are a business process modeling expert. I will provide you with a textual description of a business process. Generate a JSON model for the process. Analyze and identify key elements: 1. Start and end events. 2. Tasks and their sequence. 3. Gateways (xor or parallel) and an array of "branches" containing tasks. For xor gateways, there is a condition for the decision point and each branch has a condition label. 4. Loops: involve repeating tasks until a specific condition is met. Nested structure: The schema uses nested structures within gateways to represent branching paths. Order matters: The order of elements in the "process" array defines the execution sequence. When analyzing the process description, identify opportunities to model tasks as parallel whenever possible for optimization (if it does not contradict the user intended sequence). Use clear names for labels and conditions. Aim for granular detail (e.g., instead of "Task 1: Action 1 and Action 2", use "Task 1: Action 1" and "Task 2: Action 2"). Sometimes you will be given a previous JSON solution with user instructions to edit.
Listing 1: Instructions prompt for process generation. [8]
This component is responsible for transforming the intermediate JSON process models generated by the LLM to standard BPMN XML models. On the one hand, this is a straightforward mapping of model elements of our nested intermediate process format to the graph-structured BPMN format. However, since the intermediate format does not contain any graphical representation information, the coordinates and size of all BPMN shapes and edges are calculated and inserted by this module. This step is highly relevant for providing a good user experience in the feedback loop: A small change in the process model should also only lead to a small change in the graphical representation. Thanks to the block-structured input processes, we can compute a planar graph that deterministically includes the coordinates of the elements.
The tool has been evaluated in [8]. The evaluation data is available online 4 . Two experiments were conducted:
1. A comparison against ProMoAI [7] and a compatible prompting pattern of [4] assessing the correctness and the required numbers of tokens. 2. An evaluation with visitors of a national science fair in Austria "Lange Nacht der Forschung" (Klagenfurt, May 24th., 2024).
Experiment 1 was based on a subset of 7 processes of the PET Dataset [9]. In the experiment, the tool provided a substantial reduction of tokens of up to 94% compared to ProMoAI and showed correctness values of on average, 95% compared to 86% of the best competitor.
In the second experiment at the science fair, visitors used the tool to model processes of their own choice; 76 process models were created, and 40 visitors participated in a preliminary technology acceptance test. The experiments showed strong indications of the tool's overall usefulness and the high quality of the generated models in the feedback loop.
With the BPMN-Chatbot, we have presented a highly efficient tool for LLM-based conversational process modeling. It allows users to interactively design processes using text or voice input. It provides substantial cost reductions (up to 94%) while achieving even better levels of correctness compared to an alternative tool and a prompting strategy from the literature. We argue that such tools may drastically change the way how processes are created in the near future.
For future works, we intend to publish the tool as an open-source software that allows users to easily integrate their own prompting strategies and intermediate formats via plugins for evaluating their own approaches with users. We argue that user tests are strongly needed to evaluate the feedback loop's capabilities. We would like to take the opportunity to gather feedback from experts at the demo session. Furthermore, we plan to extend the tool to support more elements of the BPMN Meta-Model, such as pools, lanes, and data objects.