The significance of process mining lies in its ability to enable organizations to gain insights and improve eficiency by analyzing vast amounts of data generated in IT supported processes. Process mining often falls short when it comes to understanding manual processes, as it primarily captures the what but not the how of such activities. We see great potential in the growth of videos capturing various processes, which can lead to a rich source of data for process understanding and analysis, enabling a more comprehensive insight into manual workflows. However, extracting actionable insights from these videos poses significant challenges due to their unstructured nature. This doctoral thesis is about an approach that combines multimodal data analysis and extended reality (XR) techniques to enhance business process understanding. By integrating visual, textual, and audio information from videos, the proposed solution facilitates comprehensive analysis of processes, facilitating process mining, monitoring, and guidance. To address the complexities arising from an amalgamation of rich data sources, we propose three primary research objectives: (1) evaluating methods for mining relevant process information from these multimodal data sources, particularly video; (2) exploring the integration of XR technologies with enriched event logs to foster an immersive, interactive data visualization experience and accurate domain-specific modeling; and (3) determining the influence of integrating these technologies on the interpretability of process mining results. Ultimately, our study explores the way for a specialized, comprehensive approach to process mining, harnessing the power of XR for enriched event log analysis. To demonstrate the efectiveness of the proposed approach, the thesis elaborates applications in various domains, including manufacturing, logistics, healthcare, and beyond.
In an era where data is both ubiquitous and diverse, we envision a significant promise in
the proliferation of rich data sources documenting diverse processes, which can serve as
a valuable data for comprehending and dissecting processes in-depth, afording a broader
comprehension of manual workflows. One such type of data source is video, which is rich in
content but also inherently complex and high-dimensional. Process discovery goal is to take an
event log containing example behaviors and create a process model that adequately describes
the underlying process [
The rapid advancements in multimodal data analysis and extended reality technologies present
promising opportunities for the automatic extraction of valuable process-related information.
Various studies show that interdisciplinary research field of conceptual modeling and artificial
intelligence gains mutual benefits [
In the past, we saw how adding additional modality to process mining showed positive
results. Some studies showed that in fields such as humanities, social sciences and medicine
where workers follow processes and log their execution manually in textual forms instead, we
can achieve process discovery results that are very satisfactory with 88% correctly discovered
activities [
There is an imperative need to develop robust, scalable, and versatile frameworks that can seamlessly analyze multimodal data and translate them into comprehensive process understanding and guidance. This doctoral dissertation revolves around a strategy that connects the analysis of multimodal data with extended reality (XR) methods to elevate the comprehension of business processes.
The remainder of this paper is structured as follows. In Section 2, we discuss problem formulation. Following that, in Section 3, we explore the landscape of related work. Section 4 is dedicated to an examination of the planned research strategy, encompassing our proposed solution design and the pertinent techniques involved. Moving forward to Section 5, we carefully dissect the evaluation plans of our solution design and identify the key audience and beneficiaries of this thesis. In addition, we highlight and categorize the contributions of this thesis into three organized topics. Given these intended contributions, Section 6 discusses the actual research plan. Finally, in Section 7, we draw our paper to a conclusion.
The age of digital transformation has brought forward the prominence of complex processes in
various industries. Nevertheless, gaining a profound understanding of these processes, especially
from multimodal data sources like videos, remains an unresolved challenges. It is noted that in
many organizations, documentation of process knowledge is scattered around various process
information sources which introduces considerable problems [
To achieve such, we can capture the digital footprint of activities and transactions, allowing businesses to streamline their processes, identify bottlenecks, and enhance eficiency. However, it is important to recognize that not all aspects of real-world processes are adequately represented in any representation of the business process. To underscore the limitations of current Process Mining methods, let’s consider a familiar scenario – the repair process for a mobile phone. Fig. 1 represents a simplified business process for phone repair, specifically focusing on repairing a phone with a broken camera. This figure employs a flowchart format to visually map out the sequence of activities involved in the repair process, with the manual tasks highlighted in grey. This visualization aids in comprehending the steps and stages essential for conducting phone repair within a business setting, facilitating both a high-level overview and a detailed understanding of the repairment procedure. Such visual representations are valuable tools for enhancing process eficiency, quality control, and workforce training within businesses that ofer phone repair services.
Customer agreed to the repair
Perform diagnostics
Repair
Test and confirm
proper functioning
No
X Phone is repaired?
Contact the customer with final Yes cost and readiness
Customer received repaired phone
Building valuable representations of a process can be based on event logs like presented in Table 1. In this example, the event log serves as a detailed record of activities, their timestamps, sources of information, actors involved, objects in use, duration, and the respective environments in which these activities took place.
One aspect highlighted within these event logs is the source of information for each event. It is noted that most information sources are manual, indicating that a human actor manually entered data into the system to record these events. Another noteworthy element in this table is the presence of unknown timestamps, denoted by question marks, which occur when there is no interaction with a digital system to record a specific event. Consequently, this lack of digital tracking makes it challenging to determine the exact duration of certain activities, resulting in [00:28- [00:33-?] [?-00:59] 00:33]
nosis Part Order Replacement
driver nostic Tool
era Module,
? minutes
[00:5901:04]
spector
Duration Environment 10 minutes
? minutes
estimates provided as a range of minimal to maximal values. Additionally, the table reveals that tracking the objects in use during each event is a somewhat vague aspect of this process. The list of objects is not exhaustive, and it is unclear whether some objects can be used in parallel. This ambiguity can complicate resource allocation and potentially afect the eficiency of the repair process.
Integrating multimodal data sources into event logs can ofer innovative solutions to address the challenges of enriching event logs. By supplementing manual data entry with diverse data streams such as video recordings from multiple perspectives, sensor data like depth sensing maps, and audio recordings capturing specific sound events, businesses can significantly enhance the accuracy, completeness, and richness of their event logs, as illustrated in Fig. 2.
Multiple camera angles can capture diferent aspects of the repair procedure, allowing for precise and real-time documentation of technician actions and the condition of the device. Video data also ofers the advantage of providing a visual timeline, eliminating the need for uncertain timestamps. With this visual evidence, the duration of each repair step can be accurately determined, enabling more precise process analysis. Moreover, video data can be used to verify the usage of tools and objects, making it easier to track resources in parallel and optimize resource allocation. Sensor data, particularly unstructured ones, ofers a wealth of information that can complement traditional event logs. These data streams can provide insights into the intricate details of a repair, such as the depth and dimensions of components, the accuracy of alignments, and the quality of connections. Audio data recording of specific sound events can add another layer of context to event logs. For instance, the detection of particular sounds, such as clicks, whirrs, or engine sounds during the repair process, can serve as additional markers for events and their timing. These audio cues can be correlated with the corresponding visual and sensor data, ofering a holistic understanding of the repair workflow. Furthermore, audio data can assist in identifying potential issues or anomalies during the repair, facilitating real-time intervention and quality assurance. Table 2 shows sample event logs by adding multimodal data sources of information to the system. The measurable exact duration of specific activities is a crucial factor in evaluating manual processes, as it often indicates eficiency, identifies areas for improvement, and guides resource allocation. Shorter durations suggest streamlined and optimized processes, while longer durations may signal ineficiencies. Analyzing activity durations is essential for benchmarking, compliance, quality control, and ensuring a positive customer experience. However, it’s vital to consider other factors such as accuracy, quality, and process efectiveness in conjunction with timing to comprehensively evaluate manual processes, as not all processes prioritize speed above all else.
To fully harness the potential of incorporating business logic on top of enriched event logs, there is a pressing need for a specialized approach that can deal with unique demands of working with video and volumetric data in conjunction with temporal and spatial information. Such an approach may involve applications of extended reality (XR) for immersive data manipulation and precise domain-specific modeling. Incorporating XR into the analysis of enriched event logs may not only enhances the understanding of complex processes but also ofers a more intuitive and interactive approach to data analysis. To this end, three research questions emerge: [RQ1] How can we efectively mine and extract process mining-relevant information from multimodal data sources, particularly from video data? [RQ2] How can extended reality technologies be efectively integrated with enriched event logs to facilitate immersive data manipulation and visualization for precise modeling and domain-knowledge annotation? [RQ3] How does the integration of these technologies influence the interpretability of process mining results?
[00:40- [00:52-1:11] 00:52]
placement Check
era Module,
12.3 minutes
19.2 minutes
In this section, we examine related work. A broader study into related work is still ongoing, but two areas that provide promising inputs are: Immersive Process Manipulations and Video and Sensory Analysis.
There are studies on how modeling tools visualize the models, and how modelers interact with
the models [
Several scholarly papers have investigated the utility of immersive visualizations as
efective tools for immersive analytic [
While conducting a general review of immersive design approaches, as outlined in [
Oberhauser et-al previously published VR-based tools for visualizing, navigating, and
interacting with various modeling notations such as ArchiMate [
A conducted survey [
As for preservation of the process knowledge we encode models in various ways to make
the encoded models suitable for applying Machine Learning algorithms [
Understanding the semantics and meaning conveyed in instructional videos is a fundamental challenge in computer vision and natural language processing. This research stream focuses on developing approaches to automatically analyze and interpret instructional videos to extract high-level semantic information. Researchers have explored diferent methods to tackle this problem, ranging from video segmentation and action recognition to language understanding and multimodal fusion.
Some presented studies like [
In diference to our approach, none of the found approaches focuses on aligning recognised
semantics with business logic, quantifying processes into business-relevant valuable metrics,
and most importantly, eficiency of incorporating human in the loop to represent
domainexpert knowledge. We identify as a critical step in the pipeline from raw data to valuable
representations of semantics, a modeling method that follows the complexity of data that is
analysed. In our approach we aim at transforming modeling to an immersive world where
complex raw data such as 360° video data can be properly visualized. Approach from [
The topic of process mining from videos as sources is yielding very few results. In this regard, [
Our approach is filling the gap of labeling of continuous video and other multimodal sensory
data using immersive technologies and leveraging larger process-related, more detailed,
spatiotemporal incorporation of semantics. With our approach, a modeler can interact in a novel and
more optimal way with significantly more valuable data sources. In contrast to found solutions,
our approach will consider multiple conceptualizations of detected entities utilizing novel
techniques such as [
The underlying principle of event log creation in videos involves the extraction and analysis of meaningful semantic information from low-level event logs. While low-level logs capture detailed data such as object detection, sound recognition, and speech detection, the higher-level analysis focuses on incorporating the underlying semantics into these logs. This process involves identifying and categorizing events based on their semantic context, such as recognizing specific events composed of actions, identifying objects or people in diferent contexts and understanding their interactions. By incorporating semantics into the low-level event logs, the resulting higherlevel event logs provide a more comprehensive and abstract representation of the video content, enabling advanced applications like activity recognition, video summarization, and semantic search.
Scene logging involves creating a knowledge graph of entities present in videos and their
spatio-temporal changes within the video content. By constructing a knowledge graph, entities
can be represented as nodes and their relationships and interactions can be modeled as edges
connecting these nodes with additional data which may include the date, time, location, duration,
and participants of a given interaction. To capture the relationships between entities, methods
such as [
Combining data from diferent modalities To gain a comprehensive understanding of the processes, data from diferent modalities, such as visual, audio, or sensor data, can be combined. Integration techniques and data fusion algorithms can be employed to merge information from various sources, enriching the analysis and providing a more holistic view of the tracked processes.
The solution architecture depicted in Fig. 3 ofers a comprehensive, layered approach to integrating and analyzing multimodal data sources in the context of process mining. This design embodies a holistic strategy, enabling the conversion of raw data inputs into enriched conceptual models. The goal of this architecture is its applicability in a multitude of fields. The solution should show transformative impact across domains of Process Mining, where the model aids in deciphering complex processes, or Process Monitoring/Tracking, where real-time insights are extracted, or even Process Guidance, where the model provides direction and recommendations that can be valuable for both training purposes or real-time on-field guidance. Source data acquisition The foundation of our architecture begins with the Source Data Acquisition phase. This phase accumulates a wide array of input sources, ranging from videos captured from diverse vantage points, to audio recordings, in-depth sensor data maps, and detailed machine logs that track all interactions and database entries with the associated software. Given the heterogeneity of these data sources, it’s imperative to have a unified architecture to ensure that all these disparate data streams are captured, processed, and made ready for subsequent layers of analysis.
Automated contextualization (observable entity recognition) The Automated Observable Entity Recognition phase, also termed as Automated Contextualization, is predominantly AIdriven and boasts of capabilities like object recognition (initially unnamed), identification of people (who are not yet recognized as system actors), motion tracking that detects sequences of postures of individuals and objects, classifiers for predicting the categories of various resources or activities, and background isolation to discern between diverse environments and settings. The power of AI in this phase ensures that the raw data is swiftly transformed into recognizable entities and contexts, paving the way for deeper analysis and integration.
Immersive contextualization The Immersive Contextualization phase is manual and calls upon domain-knowledge experts to embed business logic into the evolving model. This immersive experience is facilitated through cutting-edge augmented reality glasses or virtual reality headsets, enabling experts to navigate through 360° videos. At this juncture, the Entity Naming process comes to the fore, serving as a verification mechanism for the automated classifications performed earlier. This is complemented by Artifact Manipulations, which provide the flexibility to either group entities into more abstract categories or further specify them. For instance, an environment can be identified as a derivative of another setting, contingent on the availability Video 1 Video 2
Audio 1 Depth Sensing
Machine Logs
Free Form Entries “This phone has a broken camera that requires repair, as it currently needs cleaning...” ...
Object 1 Object 2 Object 3
...
Action 1 Recognized Motion
Sequences Recognized Environments
Background 1
Resource Classification Object 1 - a Phone (68%) Object 1 - an Electronic device (77%) Object 2 - a Tool (98%) Object 3 - a box (84%) Object 3 - a Toy (34%) Action 1 - Screwing (53%) Environment 1 - an Office (40%) ... video or with real -time Augmented
surrounding Domain-knowledge Modeler in Extended Reality
Entity Annotation Object 1 - “Phone on repairment” : P1 Object 3 - “Toolbox 1“ : instance1 Object 2 - “Tool A“ : instance1 Object 4 - “Camera module“ : old, new Action 1 - Screwing : removingScreen Environment 1 - Work-desk1 : wd1
Artifact Manipulation Tool A Tool A.1 Tool A.2
Desk2 Relationship Annotation
using_tool_A Actor1
uses enters
RoomW in
Tool A
repairs Camera module
Event Logic Triggers if event1 uses object 1 report while event2 set status: Status on event3 save duration of a1 l e d o M l a u t p e c n o C d e c n a h n E e c n a Source Data Acquisition
Automated Contextualization Immersive Contextualization of specific tools. Relationship Annotation further enriches the model, charting out a graph of pertinent links between artifacts and activities. The culmination of this phase is the Event Triggers module, where intricate business logic rules and advanced connections between the recognized entities and activities are established.
Enhanced conceptual model After traversing through the intricate layers of the architecture, what emerges is the Enhanced Conceptual Model. This model is not only aware of the enriched multimodal event log but is also primed for diverse applications. It encapsulates the depth and breadth of the information from the varied data sources and the insights garnered through automated and manual contextualization.
The foundation of our architecture begins with the Source Data Acquisition phase. This section accumulates a wide array of input sources, ranging from videos captured from diverse vantage points, to audio recordings, in-depth sensor data maps, and detailed machine logs that track all interactions and database entries with the associated software. Given the heterogeneity of these data sources, it’s imperative to have a unified architecture to ensure that all these disparate data streams are captured, processed, and made ready for subsequent layers of analysis. 4.4. Automated contextualization (observable entity recognition) Following the data collection phase, the architecture delves into the Automated Observable Entity Recognition section, also termed as Automated Contextualization. This segment is predominantly AI-driven and boasts of capabilities like object recognition (initially unnamed), identification of people (who are not yet recognized as system actors), motion tracking that detects sequences of postures of individuals and objects, classifiers for predicting the categories of various resources or activities, and background isolation to discern between diverse environments and settings. The power of artificial intelligence in this phase ensures that the raw data is swiftly transformed into recognizable entities and contexts, paving the way for deeper analysis and integration.
The subsequent tier, Immersive Contextualization, is manual and calls upon domain-knowledge experts to embed business logic into the evolving model. This immersive experience is facilitated through cutting-edge augmented reality glasses or virtual reality headsets, enabling experts to navigate through 360-degree videos. At this juncture, the Entity Naming process comes to the fore, serving as a verification mechanism for the automated classifications performed earlier. This is complemented by Artifact Manipulations, which provide the flexibility to either group entities into more abstract categories or further specify them. For instance, an environment can be identified as a derivative of another setting, contingent on the availability of specific tools. Relationship Annotation further enriches the model, charting out a graph of pertinent links between artifacts and activities. The culmination of this phase is the Event Triggers module, where intricate business logic rules and advanced connections between the recognized entities and activities are established.
After traversing through the intricate layers of the architecture, what emerges is the Enhanced Conceptual Model. This model is not only aware of the enriched multimodal event log but is also primed for diverse applications. It encapsulates the depth and breadth of the information from the varied data sources and the insights garnered through automated and manual contextualization.
5.1. Identification of the target audience and beneficiaries of the thesis This thesis primarily targets industries that experience gaps or “blind spots” in their event logs due to manual or physically-based processes. Industries like manufacturing, logistics, warehousing, agriculture, and construction, which heavily rely on manual labor and physical operations, often lack comprehensive electronic tracking of every nuanced event. The proposed approach for extracting information from multimodal data sources, especially video data, ofer a groundbreaking way for these sectors to digitize and analyze manual processes that were previously opaque. Extended reality technologies can provide these industries with immersive training and process understanding tools, making the shift from manual oversight to digital monitoring smoother and more intuitive. Additionally, with the inclusion of BPMN in multimodal data analysis, businesses can gain contextually richer insights, aiding in process optimization. Operational managers in these sectors can not only interpret the newly available process mining results but also predict future process bottlenecks and constraints. Furthermore, organizations aiming to create a standardized blueprint of their manual processes can leverage the business process libraries proposed by this research. All in all, this thesis promises a transformational shift for industries traditionally limited by manual process constraints.
To ensure the robustness and applicability of the approach proposed for the planned thesis, it is imperative to employ rigorous validation techniques. One such technique is prototyping. Given the novel amalgamation of video, audio, and sensor data with XR technologies, creating a prototype ofers a tangible representation of our concepts and allows stakeholders to interact with and refine the proposed system. By ofering an early visualization of the system’s functionalities and interfaces, prototyping facilitates immediate feedback, highlights potential pitfalls, and ofers avenues for iterative refinement. Moreover, a prototype can simulate the data integration process, giving researchers a glimpse into the challenges and solutions of working with disparate data types in a unified XR environment.
Additionally, field experiments stand out as a powerful validation approach for our research. By applying our methods in real-world settings, we can assess the practicality, eficiency, and accuracy of our proposed techniques. Observing domain-experts as they engage with our XR-powered process mining tool can provide valuable insights into its usability, efectiveness, and areas for improvement. Such experiments also help in determining the tool’s impact on business process workflows, stakeholders satisfaction, and the overall quality of monitored processes. Meanwhile, argumentation, although more theoretical, ofers a structured platform to articulate and defend the reasoning behind our approach, ensuring that the choices made in data integration, XR application, and process analysis are not only innovative but also logically sound and consistent with existing domain knowledge.
The contribution of the thesis follows three topics: Topic 1 – Mining: Focused on extracting meaningful information from data. It handles the visualization of data, manages multi-view video data, detects process loops, visualizes dependencies, identifies critical paths, and more.
Topic 2 – Tracking and monitoring: Uses videos, other modalities and extended reality for real-time process tracking, ensuring continuity of processes. It can detect anomalies, handle variations in processes over time, and create reports like summaries.
Topic 3 – Guiding: Utilizes extended reality for process guidance, transforming data into interactive guides. The topic also delves into data visualization techniques, user-friendly interface design, and training requirements.
In our journey thus far, we have explored and tested an array of technologies, harnessing their potential for process mining in an enriched multimodal environment. Our current explorations have centered around fine-tuning pre-trained models tailored for both visual and audio recognition. Recognizing the power of visual language models, we conducted experiments to mine business logic from video frames. By furnishing these models with tailored prompts, we’ve made significant strides in extracting meaningful insights from the visual data. Additionally, our tests of human body pose estimation, image segmentation, and image classification have further equipped us with tools to understand and interpret the vast and varied visual inputs at our disposal.
Looking ahead, our roadmap is illustrated in Fig. 4. Our immediate endeavor is the development of an XR tool designed to empower modelers in navigating 360° videos, allowing them to pinpoint and select relevant objects with precision. This selection process is envisioned to be intuitive and immersive, leveraging a cube selector to volumetrically extract the mesh of the scene. This hands-on approach ensures that the extracted data is both relevant and contextual. Once this foundational tool is in place, our energies will shift toward the creation of a comprehensive immersive modeling tool. This tool will be pivotal in preparing us for the subsequent phase of data labeling, ensuring that the raw data is categorized and marked for further analysis. As we progress, the collection of source data will be paramount, and we anticipate diving deeper into state-of-the-art pre-trained models that can assist in AI-driven contextualizations.
The proposed solution is designed to address a multitude of challenges and considerations across various domains. One critical aspect is ensuring interoperability with diferent data formats. This entails the integration, standardization, and compatibility of diverse data inputs, allowing the solution to handle data from various sources efectively. This feature is pivotal in enabling the solution to be adaptable and versatile in processing information. Another essential feature focuses on accommodating diverse user needs. By ofering customization options, user interface personalization, and adaptive features, the solution aims to enhance user-friendliness and satisfaction. This approach ensures that users from diferent backgrounds and with varying requirements can efectively utilize the solution to meet their specific needs.
Handling biases in multimodal data is another crucial aspect addressed in the solution. It explores methods for detecting, mitigating, and assessing biases, which is essential for
Problem statement and testing objectives Immersive modeling of multimodal event logs AI-assisted video event log creation AI-assisted multimodal event log creation Improving interpretability of process mining
Exploring application domains for process mining, tracking and guidance maintaining the integrity and reliability of analysis results. This feature underscores the commitment to fairness and objectivity in data analysis. Dealing with missing or incomplete data is a common challenge in data analysis, and the solution provides strategies for addressing this issue. Techniques such as imputation, data augmentation, and sensitivity analysis are explored to mitigate the impact of missing information on analysis results, ensuring more robust outcomes.
Concluding our roadmap, our focus will pivot towards diverse applications for our integrated system. Moreover, as we venture deeper into process mining, ensuring the interpretability of these mined processes will be crucial. This will ensure that the insights gleaned from our system are not only accurate but also actionable, lending them real-world relevance and applicability.
The future work can dive into advanced topics like real-time monitoring, faster data analysis, and accelerated process improvement, showcasing the potential for innovation in data analysis methods. Additionally, future work considers eficient storage strategies, concurrent process mining, uncovering hidden process patterns, predicting process performance improvement, and structuring business process libraries, highlighting a comprehensive approach to data analysis and process optimization. The solution’s ability to be applied successfully in various industries is a testament to its adaptability and potential for widespread utility.
Our investigation into the integration of multimodal data sources and extended reality technologies in process mining has illuminated the transformative potential of this synergistic approach. As demonstrated in our solution design architecture, the journey from Source Data Acquisition to an Enhanced Conceptual Model underscores the power of combining raw data inputs with both automated and immersive manual contextualization techniques. The goal of this integrated approach is not only in its capacity to transform heterogeneous data into actionable insights but also in its versatility, ofering applications across process mining, monitoring, and guidance domains.
The implications of this research are profound for industries reliant on process analyses, especially those in repair and maintenance sectors. Our architectural model ofers organizations a blueprint to harness their diverse data streams, contextualize them through AI and human expertise, and subsequently derive enhanced, actionable conceptual models.
In summation, while the horizon of process mining and analysis is expansive, our study introduces a pioneering approach that marries the best of technology and human expertise. Future endeavors in this field should focus on scalability, ensuring that our architecture can cater to even more complex and voluminous data environments, and further bridging the gap between raw data and actionable insights.
Authors are deeply grateful to Prof. Dr. Henrik Leopold at Kühne Logistics University (KLU) for his invaluable guidance in shaping this work.