=Paper=
{{Paper
|id=Vol-3783/paper_352
|storemode=property
|title=Journey-to-Process Analytics: Fusing Experience with Operations Data
|pdfUrl=https://ceur-ws.org/Vol-3783/paper_352.pdf
|volume=Vol-3783
|authors=Alexander Rochlitzer,Manuel Meindl,Raheleh Hadian,Gregory Hakomaki II,Henrik Leopold
|dblpUrl=https://dblp.org/rec/conf/icpm/RochlitzerMHHL24
}}
==Journey-to-Process Analytics: Fusing Experience with Operations Data==
https://ceur-ws.org/Vol-3783/paper_352.pdf
Journey-to-Process Analytics: Fusing Experience with
Operations Data
Alexander Rochlitzer1,∗ , Manuel Meindl2 , Raheleh Hadian2 , Gregory Hakomaki II2
and Henrik Leopold1
1
Kühne Logistics University, Großer Grasbrook 17, 20457 Hamburg, Germany
2
SAP, George-Stephenson-Straße 7-13, 10557 Berlin, Germany
Abstract
A key business challenge of process mining is to appeal to decision-makers who seek to differentiate,
with the ambition to go beyond operational optimization. One way to position process mining as a
differentiator is to integrate operational process and experience journey perspectives, with the ultimate
goal to better align operations with the needs of customers and other external stakeholders. To exemplify
this direction, this demonstration presents SAP’s journey-to-process analytics capabilities that fuse
experience with process data, allowing organizations to generate insights about how operations affect
experience.
Keywords
Process Mining, Business Process Improvement, Process Observability, Large Language Models
Metadata description Value
Tool name Journey-to-Process Analytics
Current version 1.0
Legal code license Proprietary
Languages, tools and services used Python, Kubernetes, Docker, SAP Data Custodian, ReactJS
Supported operating environment Microsoft Windows, GNU/Linux
Download/Demo URL https://staging.signavio.com/g/statics/labs/journey2process
Documentation URL https://url.sap/7o0crn
Source code repository N/A
Screencast video https://url.sap/vm93si
1. Introduction
To thrive in today’s fast-moving, interconnected world, organizations must not only maintain
tight control over their processes but also continuously transform them to meet the needs of
their stakeholders. Traditional business process management (BPM), however, focuses on an
organization’s inside-out perspective, thereby missing valuable improvement opportunities that
arise from leveraging the knowledge and experiences of stakeholders [1], who, after all, define
ICPM 2024 Tool Demonstration Track, October 14-18, 2024, Kongens Lyngby, Denmark
∗
Corresponding author.
Envelope-Open alexander.rochlitzer@klu.org (A. Rochlitzer); manuel.meindl@sap.com (M. Meindl); raheleh.hadian@sap.com
(R. Hadian); gregory.hakomaki.ii@sap.com (G. Hakomaki II); henrik.leopold@klu.org (H. Leopold)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Figure 1: Text analysis dashboard showing the expert-defined process topics and sentiment insights
the success of an organization. One of the main challenges of leveraging the stakeholders’
perspective is that stakeholder journey data are often unstructured (e.g., survey comments,
incident tickets, emails), meaning that extracting relevant information and combining it with
operational data may require significant manual effort.
Recognizing both the lack of process observability and the recent interest in Large Language
Models (LLM) for BPM [2, 3, 4], we propose a natural language-driven tool that analyzes journey
data and maps them to process data and models, enabling a more holistic view on organizational
operations. The tool provides organizations with in-depth insights into how their stakeholders
perceive individual process steps or stages. These insights can help enhance the understanding
of the process dynamics that affect the experience of stakeholders, speed up the search for
necessary changes that address the root causes of experience issues, and facilitate the validation
of the changes’ actual impact on the experience.
The remainder of the paper is structured as follows. Section 2 presents our tool for the
integrated analysis of journey and process data. Section 3 elaborates on the maturity of the
tool. Section 4 outlines directions for future work and concludes the paper.
2. Architecture and Features
The tool provides two main functionalities for leveraging the journey perspective for BPM: the
textual analysis of journey data and the mapping of journey data to process data and models.
Below, we elaborate on the details of these functionalities.
2.1. Journey Data Analysis
The journey-to-process analytics tool starts with the analysis of journey data as the cornerstone
of providing an outside-in perspective. Users first need to create a new dashboard and provide a
CSV file containing the data to be analyzed. The tool then determines the sentiment of the data,
classifies the data into process topics, and identifies groups of similar stakeholder experiences.
Sentiment Analysis. To get insights into how stakeholders perceive the processes of an
organization, the tool classifies each text of the stakeholders as either positive, negative, neutral,
or unknown. Users can then filter by sentiment type and review each text individually. Alterna-
tively, users can analyze the development of the overall sentiment of the journey data over time,
as shown in Figure 1. The tool provides the overall sentiment both as a net sentiment score and
broken down into the sentiment types. The net sentiment score 𝑆net is defined as follows:
𝑁pos − 𝑁neg
𝑆net =
𝑁pos + 𝑁neg + 𝑁neu
where 𝑁pos is the number of positive items, 𝑁neg is the number of negative items, 𝑁neu is the
number of neutral items
To determine the sentiment of a stakeholder’s text, we use an LLM that receives a prompt
that includes the text and the sentiment types. A text that might violate the LLM’s content
restrictions is classified as unknown.
Process Topic Analysis. To provide a starting point for a targeted in-depth analysis by the
user, the tool classifies the data according to six expert-defined process topics, as shown in
Figure 1. To help users to identify process issues that require attention, the tool provides, for
each topic, a sentiment score and a brief summary for each sentiment type. Based on this
information, users can drill down into relevant topics. Table 1 provides a description of each
topic and a corresponding example.
We implemented this analysis as a multi-label classification task, where each text from a
stakeholder can be associated with zero or more process topics. To obtain the topics for a text,
we provide an LLM with a prompt that includes the process topics, their descriptions and the
stakeholder’s text. In addition, we generate a summary by using a prompt that takes in texts
from the specific topic and sentiment type for which the summary is generated.
Cluster Analysis. Similar to the classification of journey data into predefined topics, the tool
classifies the data into ”What goes well” and what ”Needs to Improve”, as shown in Figure 2.
Within these two categories, the tool provides more fine-granular topics that emerge from
the specific experience data. To achieve this, we use the BERTopic library1 for clustering
the data. For each emerging cluster, our tool also provides a topic label and a summary.
We generate them using an LLM with a prompt that leverages representative keywords
and texts according to BERTopic from the cluster for which the label and summary are generated.
As this functionality does not require process data or models, it enables even organi-
zations with very low levels of process management maturity to get an outside-in perspective
on their operations.
1
https://maartengr.github.io/BERTopic
Process Topic Description Example
Process Clarity Any reference by the subject to the I was not sure how to approve my
understanding of the process that team’s targets.
feedback is given to.
Process Efficiency Any reference to the quality of exe- I always have to go back and forth in
cution of a process, with regards to the system to approve the targets of
costs, effort, time. each team member.
Process Speed Any reference to speed regarding a My team lead approved my targets
process. Explicit subcategory of pro- quickly.
cess efficiency.
Process Effectiveness Any reference to the success of the I joined in the middle of the year and
outcome of a process in one instance I could not set any targets for the re-
of execution or the overall success of maining months.
the process and its alignment to the
overall goal.
System related Any reference to the systems, tools The target setting system is easy to use.
and integrations or UI and UX serv-
ing as an underlying layer to a pro-
cess.
Human related Any reference to people serving as My team lead was very nice and pro-
stakeholders in a process. vided constructive feedback when we
discussed my targets.
Table 1
Description of the six expert-defined process topics
2.2. Journey-to-Process Mapping
To combine the outside-in and inside-out perspectives into a holistic view, users can link a
process model (BPMN file) or an event log (XES file) to their provided journey data. The tool
then maps each text of a stakeholder to zero or more activities of the model or nodes of the mined
directly-follows graph (DFG). The mapping dashboards provide an overview of the analyzed
process with sentiment scores attached to each process step. As shown in Figure 2, users can
also select a particular step to get more in-depth insides about how stakeholders perceive this
step, including a breakdown of the sentiment into the different types. Similar to the analysis of
process topics, the mapping of journey data to process data and models is implemented as a
multi-label classification task. To obtain the relevant process elements for a stakeholder’s text,
we provide a prompt that includes the activities or nodes and the text to an LLM.
3. Maturity
In this section, we report on the evaluation results of the novel process topic analysis and
journey-to-process mapping features. For the topic matching experiment, we used a synthetic
dataset that is inspired by real-life customer and employee journey data of a variety of business
scenarios, including performance review, hiring, and order-to-cash processes. The dataset
Figure 2: Event log dashboard showing a directly-follows graph with the sentiment for a specific activity
(left) and the feedback clusters (right)
consists of 100 feedback comments. For the evaluation of the activity matching feature, we
used a synthetic dataset inspired by the real-life order-to-cash process mentioned above. The
dataset consists of 16 process activities and 57 feedback comments. To evaluate the features’
performance, we used the average dice score. The dice score 𝑆dice is defined as follows:
2 ⋅ |𝑋 ∩ 𝑌 |
𝑆dice =
|𝑋 | + |𝑌 |
where 𝑋 is the set of human-provided labels, 𝑌 is the set of predicted labels, |𝑋 ∩ 𝑌 | is the size
of the intersection between sets 𝑋 and 𝑌, |𝑋 | is the size of set 𝑋, |𝑌 | is the size of set 𝑌
The process topic analysis feature and the journey-to-process mapping feature experiments
were performed in a one-shot setting and a zero-shot setting, respectively. For both experiments,
we used GPT-4 [5] as the LLM. This resulted in average dice scores of 0.71 and 0.5 for the process
topic analysis and journey-to-process mapping, respectively. Furthermore, we conducted case
studies with companies from various industries and found that the tool can help to validate
business assumptions before taking actions, to align key performance indicators with stakehold-
ers’ experiences, and to perform more targeted actions such as training sessions for a specific
group of employees or process changes for a specific customer type. In addition, we found that
the journey-to-process mapping can be significantly enhanced by human-in-the-loop feedback
about the validity of the mappings, allowing the tool to adjust to user specific terminologies.
4. Conclusion and Future Work
The journey-to-process analytics tool enables organizations to perform an integrated analysis of
journey and operational data, providing a more holistic and actionable view on organizational
operations. The tool analyzes the sentiment of journey data, classifies the data into expert-
defined process topics, identifies groups of similar experiences, and provides in-depth insights
into how stakeholders perceive individual process steps or stages. The increased process
observability enables organizations to make informed and timely adjustments. By performing
experiments and elaborating on case studies, we demonstrated the effectiveness of the tool.
In the future, we aim to enable users to adapt the tool to their data by confirming, rejecting,
or modifying the sentiment assessments of their journey data and the associations of the data
with process topics and process elements. Additionally, we aim to provide a more granular
sentiment analysis that determines the sentiment towards each individual element addressed in
a journey data point. Finally, the tool’s mapping functionality can be improved by considering
more element types, by mapping journey data collected in a process-oriented manner to event
logs instead of DFGs, and by providing a two-step mapping process. In the first step, the process
model or event log that is related to the journey data would be identified within a collection. In
the second step, the journey data would be mapped to individual process elements.
Acknowledgments
We sincerely thank Timotheus Kampik and Peyman Badakhshan for their continuous support
in this research project and for revising an earlier draft of this manuscript.
References
[1] M. Rosemann, Proposals for future bpm research directions, in: C. Ouyang, J.-Y. Jung (Eds.),
Asia Pacific Business Process Management, Springer International Publishing, Cham, 2014,
pp. 1–15.
[2] T. Kampik, C. Warmuth, A. Rebmann, R. Agam, L. N. P. Egger, A. Gerber, J. Hoffart, J. Kolk,
P. Herzig, G. Decker, et al., Large process models: A vision for business process management
in the age of generative ai, KI - Künstliche Intelligenz (2024).
[3] K. Busch, A. Rochlitzer, D. Sola, H. Leopold, Just tell me: Prompt engineering in business
process management, in: H. van der Aa, D. Bork, H. A. Proper, R. Schmidt (Eds.), Enterprise,
Business-Process and Information Systems Modeling, Springer Nature Switzerland, Cham,
2023, pp. 3–11.
[4] M. Vidgof, S. Bachhofner, J. Mendling, Large language models for business process manage-
ment: Opportunities and challenges, in: C. Di Francescomarino, A. Burattin, C. Janiesch,
S. Sadiq (Eds.), Business Process Management Forum, Springer Nature Switzerland, Cham,
2023, pp. 107–123.
[5] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, et al., Gpt-4 technical report (2023). arXiv:2303.08774 .