To thrive in today's fast-moving, interconnected world, organizations must not only maintain tight control over their processes but also continuously transform them to meet the needs of their stakeholders. Traditional business process management (BPM), however, focuses on an organization's inside-out perspective, thereby missing valuable improvement opportunities that arise from leveraging the knowledge and experiences of stakeholders [1], who, after all, define ICPM 2024 Tool Demonstration Track, October 14-18, 2024, Kongens Lyngby, Denmark * Corresponding author. Envelope alexander.rochlitzer@klu.org (A. Rochlitzer); manuel.meindl@sap.com (M. Meindl); raheleh.hadian@sap.com (R. Hadian); gregory.hakomaki.ii@sap.com (G. Hakomaki II); henrik.leopold@klu.org (H. Leopold) the success of an organization. One of the main challenges of leveraging the stakeholders' perspective is that stakeholder journey data are often unstructured (e.g., survey comments, incident tickets, emails), meaning that extracting relevant information and combining it with operational data may require significant manual effort.
Recognizing both the lack of process observability and the recent interest in Large Language Models (LLM) for BPM [2,3,4], we propose a natural language-driven tool that analyzes journey data and maps them to process data and models, enabling a more holistic view on organizational operations. The tool provides organizations with in-depth insights into how their stakeholders perceive individual process steps or stages. These insights can help enhance the understanding of the process dynamics that affect the experience of stakeholders, speed up the search for necessary changes that address the root causes of experience issues, and facilitate the validation of the changes' actual impact on the experience.
The remainder of the paper is structured as follows. Section 2 presents our tool for the integrated analysis of journey and process data. Section 3 elaborates on the maturity of the tool. Section 4 outlines directions for future work and concludes the paper.
The tool provides two main functionalities for leveraging the journey perspective for BPM: the textual analysis of journey data and the mapping of journey data to process data and models. Below, we elaborate on the details of these functionalities.
The journey-to-process analytics tool starts with the analysis of journey data as the cornerstone of providing an outside-in perspective. Users first need to create a new dashboard and provide a CSV file containing the data to be analyzed. The tool then determines the sentiment of the data, classifies the data into process topics, and identifies groups of similar stakeholder experiences.
Sentiment Analysis. To get insights into how stakeholders perceive the processes of an organization, the tool classifies each text of the stakeholders as either positive, negative, neutral, or unknown. Users can then filter by sentiment type and review each text individually. Alternatively, users can analyze the development of the overall sentiment of the journey data over time, as shown in Figure 1. The tool provides the overall sentiment both as a net sentiment score and broken down into the sentiment types. The net sentiment score 𝑆 net is defined as follows:
where 𝑁 pos is the number of positive items, 𝑁 neg is the number of negative items, 𝑁 neu is the number of neutral items
To determine the sentiment of a stakeholder's text, we use an LLM that receives a prompt that includes the text and the sentiment types. A text that might violate the LLM's content restrictions is classified as unknown.
Process Topic Analysis. To provide a starting point for a targeted in-depth analysis by the user, the tool classifies the data according to six expert-defined process topics, as shown in Figure 1. To help users to identify process issues that require attention, the tool provides, for each topic, a sentiment score and a brief summary for each sentiment type. Based on this information, users can drill down into relevant topics. Table 1 provides a description of each topic and a corresponding example.
We implemented this analysis as a multi-label classification task, where each text from a stakeholder can be associated with zero or more process topics. To obtain the topics for a text, we provide an LLM with a prompt that includes the process topics, their descriptions and the stakeholder's text. In addition, we generate a summary by using a prompt that takes in texts from the specific topic and sentiment type for which the summary is generated.
Cluster Analysis. Similar to the classification of journey data into predefined topics, the tool classifies the data into "What goes well" and what "Needs to Improve", as shown in Figure 2. Within these two categories, the tool provides more fine-granular topics that emerge from the specific experience data. To achieve this, we use the BERTopic library1 for clustering the data. For each emerging cluster, our tool also provides a topic label and a summary. We generate them using an LLM with a prompt that leverages representative keywords and texts according to BERTopic from the cluster for which the label and summary are generated.
As this functionality does not require process data or models, it enables even organizations with very low levels of process management maturity to get an outside-in perspective on their operations.
Process Clarity Any reference by the subject to the understanding of the process that feedback is given to.
I was not sure how to approve my team's targets.
Any reference to the quality of execution of a process, with regards to costs, effort, time.
I always have to go back and forth in the system to approve the targets of each team member.
Any reference to speed regarding a process. Explicit subcategory of process efficiency.
Process Effectiveness Any reference to the success of the outcome of a process in one instance of execution or the overall success of the process and its alignment to the overall goal.
I joined in the middle of the year and I could not set any targets for the remaining months.
Any reference to the systems, tools and integrations or UI and UX serving as an underlying layer to a process.
The target setting system is easy to use.
Any reference to people serving as stakeholders in a process.
Description of the six expert-defined process topics
To combine the outside-in and inside-out perspectives into a holistic view, users can link a process model (BPMN file) or an event log (XES file) to their provided journey data. The tool then maps each text of a stakeholder to zero or more activities of the model or nodes of the mined directly-follows graph (DFG). The mapping dashboards provide an overview of the analyzed process with sentiment scores attached to each process step. As shown in Figure 2, users can also select a particular step to get more in-depth insides about how stakeholders perceive this step, including a breakdown of the sentiment into the different types. Similar to the analysis of process topics, the mapping of journey data to process data and models is implemented as a multi-label classification task. To obtain the relevant process elements for a stakeholder's text, we provide a prompt that includes the activities or nodes and the text to an LLM.
In this section, we report on the evaluation results of the novel process topic analysis and journey-to-process mapping features. For the topic matching experiment, we used a synthetic dataset that is inspired by real-life customer and employee journey data of a variety of business scenarios, including performance review, hiring, and order-to-cash processes. The dataset consists of 100 feedback comments. For the evaluation of the activity matching feature, we used a synthetic dataset inspired by the real-life order-to-cash process mentioned above. The dataset consists of 16 process activities and 57 feedback comments. To evaluate the features' performance, we used the average dice score. The dice score 𝑆 dice is defined as follows:
where 𝑋 is the set of human-provided labels, 𝑌 is the set of predicted labels, |𝑋 ∩ 𝑌 | is the size of the intersection between sets 𝑋 and 𝑌, |𝑋 | is the size of set 𝑋, |𝑌 | is the size of set 𝑌
The process topic analysis feature and the journey-to-process mapping feature experiments were performed in a one-shot setting and a zero-shot setting, respectively. For both experiments, we used GPT-4 [5] as the LLM. This resulted in average dice scores of 0.71 and 0.5 for the process topic analysis and journey-to-process mapping, respectively. Furthermore, we conducted case studies with companies from various industries and found that the tool can help to validate business assumptions before taking actions, to align key performance indicators with stakeholders' experiences, and to perform more targeted actions such as training sessions for a specific group of employees or process changes for a specific customer type. In addition, we found that the journey-to-process mapping can be significantly enhanced by human-in-the-loop feedback about the validity of the mappings, allowing the tool to adjust to user specific terminologies.
The journey-to-process analytics tool enables organizations to perform an integrated analysis of journey and operational data, providing a more holistic and actionable view on organizational operations. The tool analyzes the sentiment of journey data, classifies the data into expertdefined process topics, identifies groups of similar experiences, and provides in-depth insights into how stakeholders perceive individual process steps or stages. The increased process observability enables organizations to make informed and timely adjustments. By performing experiments and elaborating on case studies, we demonstrated the effectiveness of the tool.
In the future, we aim to enable users to adapt the tool to their data by confirming, rejecting, or modifying the sentiment assessments of their journey data and the associations of the data with process topics and process elements. Additionally, we aim to provide a more granular sentiment analysis that determines the sentiment towards each individual element addressed in a journey data point. Finally, the tool's mapping functionality can be improved by considering more element types, by mapping journey data collected in a process-oriented manner to event logs instead of DFGs, and by providing a two-step mapping process. In the first step, the process model or event log that is related to the journey data would be identified within a collection. In the second step, the journey data would be mapped to individual process elements.
We sincerely thank Timotheus Kampik and Peyman Badakhshan for their continuous support in this research project and for revising an earlier draft of this manuscript.