Process mining is a family of techniques to analyze business processes based on the event logs produced during their execution. While most process mining techniques rely on readily available event logs, extracting them poses challenges and limits their availability and usability. As event-driven architecture is gaining momentum, event streaming platforms like Apache Kafka can solve this problem. This paper proposes an architecture that allows using Kafka both as message broker for the running process as well as data store directly enabling process mining, including streaming process mining on the same dataset.
Process mining is a family of techniques to analyze business processes based on event logs
produced during their execution [
While process mining techniques rely on event logs, extracting them from the information
systems supporting the process is not trivial. If a process is orchestrated by a BPMS, event data
can be easily extracted in the right format [
Streaming process mining refers to a set of techniques and tools dealing with processing
streaming event data [
Event streaming is the practice of capturing data in real-time from diferent sources in form
of streams of events, storing these streams durably for later retrieval, processing and reacting
to these streams in real-time and retrospectively, as well as routing these events to specified
destinations [
An event in Kafka has a key, value, timestamp and optional metadata. The events are organized and stored in topics. A topic can have multiple producers and multiple consumers. In contrast to traditional messaging systems, the events are not deleted after consumption. This allows multiple consumers to process the events at their own pace, as well as read the entire topic from the beginning. For each topic, the retention period after which the events are deleted can be set. Importantly, this retention period can also be set to infinity, allowing for permanent storage of events. To allow for scalability, topics are split into partitions, which are scattered across multiple brokers. Events with the same key are always written to the same partitions, and Kafka guarantees the order preservation for the events in the same partition.
This section presents the architecture for a Kafka-based system supporting process mining. Section 3.1 gives an overview of the system architecture. Section 3.2 describes the topic architecture, i.e. how events are split into topics. Section 3.3 describes the proposed serialization format of events in Kafka. Section 3.4 discusses the role of a BPMS in the proposed architecture, and Section 3.5 explains how process mining can be performed.
The proposed system architecture is shown in Figure 1. The central element is the Kafka event streaming platform that connects the other components. Importantly, Kafka serves as a communication backbone for the Services responsible for executing the tasks in the process. Depending on the environment where such system is deployed, there might already be a BPMS present, in which case it also should communicate with Kafka. Essentially, Kafka serves as the single source of truth for all traditional process mining components and for streaming process mining, which might have diferent data requirements.
In Kafka, event streams are stored in topics. The number and semantics of the topics is not predefined and should be selected according to the domain specifics. This consideration is crucial since Kafka only guarantees order preservation for events within one partition of the same topic. Thus, the events that need strict order, e.g. events within one case, should be stored in one partition. This is achieved by assigning the same key to all these events, e.g. a case ID. In this section, four strategies are proposed.
Case ID as topic. In this case, for each case that has started, a new topic has to be created in the broker. All further events referencing this case ID should be sent to this topic. Consumers should regularly poll the broker for new topics and subscribe to new topics that appear. Each case topic should have only one partition to ensure the event order within the case. Such architecture would, however, sufer from a number of other issues. First, the existing system supporting the process will have to be re-designed to accommodate for the new topics. Also, having such number of topics will introduce unnecessary overhead, especially as the number of old inactive topics grows. Finally, while such architecture is very much oriented on traditional event logs with single case ID, object-centric logs are not supported.
Activity as topic. This architecture most closely resembles known microservice best practices. Each microservice (or other system) responsible for a task writes the corresponding events in its topic. The events are later picked up by the services relying on them. Case IDs can be used as event keys, allowing for simple partitioning. For object-centric processes, the object ID of some object involved in the activity can be used. A benefit of such strategy is the minimality of change necessary in the system if it already followed event-driven architecture. In the ideal case, the system can remain untouched (except the schema update, see Section3.3) and additional consumers can be seamlessly added. Events having the same key (case or object ID) will benefit from consistent partitioning and total order within partitions. A downside of such system is, however, the partial view that every topic would have on the execution. The streams will have to be aggregated (using Kafka Streams API or processing outside of Kafka), and event correlation will be crucial. However, it might be solved with the event keys and timestamps provided by Kafka producer or broker.
Process as topic. Another approach is to store all events related to one business process in one topic. Case IDs will be used as keys for the traditional processes. The keys for the processes with no case ID are subject to discussion, however, it is reasonable to use the ID of some object and perform event correlation outside Kafka. The activity label must be included in the event value, and the consuming services will not only have to monitor the right topic but also make sure to only process events with the correct activity label. A clear benefit is that this setting is the closest to the desired output event log. Each topic will simply contain all executions of a process and can almost directly be used in process mining. A downside is the additional overhead on the (micro-)services that will have to process all events and filter them based on the activity label.
Single topic. A variation of the previous strategy, which can be especially useful in scenarios where borders of business processes are unclear. In this case, all events are put into the same topic. The event key for partitioning has to be defined using domain knowledge. A benefit of such strategy is the simplicity of implementation, as it only minimally uses Kafka as a universal message broker. However, in this setup the (micro-)services will sufer from even more filtering overhead. In addition, such single topic can quickly become a data swamp, especially if infinite retention is selected.
Kafka itself is schema-less and stores the events as raw bytes, however, it is good practice to
use common data exchange formats like JSON2 , protobuf3 or Avro4 . In process mining, on the
other hand, XES5 and OCEL [
This paper proposes using JXES and OCEL JSON as the event formats. It must be noted that OCEL objects cannot be stored in the event streams but rather will have to be reconstructed from the events that created or referenced used them at the event log construction stage.
The payload, or the value of the events should contain all the necessary event attributes. For the traditional logs, it is recommended to store at least concept:name, case ID attribute and time:timestamp of the producing application. While some of the attributes can be inferred from Kafka events depending on the chosen topic architecture, it improves the readability and unifies the approach regardless of the topic architecture. Note that the case ID on the event level has no standard attribute in XES, thus a custom one, e.g. caseid has to be used. Additional attributes should be stored in the same way as in XES event logs. For the object-centric processes, in addition, the referenced objects with their IDs and relevant attributes must be included in the event value. While not all of the stored attributes are necessary for the services processing the events, e.g. case ID, such services should be configured to simply copy these attributes to their output events.
The proposed architecture can be used both in conjunction with a BPMS as well as without it. It can be distinguished between the following deployment scenarios.
BPMS-centric. In these scenarios, BPMS plays a central role in orchestrating a process. The events have a single case identifier, and external services are heterogeneous, requiring not only event-driven communication but also other methods, such as REST, Long polling and Java API. In these scenarios, a BPMS indeed serves as a single source of truth for process mining, and it would be challenging to meaningfully extract the process data from other sources. However, the events can be constantly extracted from the BPMS into Kafka to enable streaming process mining with minimal system changes.
BPMS-aided. If a BPMS is used to orchestrate a fully-automated process with event-driven services, this theoretically means both the BPMS and Kafka can be used as the primary event store. However, it may be still beneficial to use the proposed event serialization format and topic architecture to of-load data extraction tasks to Kafka. In addition, such scenarios can profit from streaming process mining if the proposed architecture is used.
No-BPMS. For other scenarios, where no BPMS is present, the proposed architecture becomes crucial for enabling process mining. If the process is composed of loosely coupled (micro-)services and systems, having a unified source of data is the only alternative to the complicated and error-prone process of extracting and correlating events (with possibly incomplete information) from heterogeneous internal storages of the respective systems.
Infinite retention allows to use Kafka both as message broker for the automated services as well as storage for historic data. The proposed architecture directly enables process mining on 6https://www.ocel-standard.org/specification/formats/json/ event streams. The last component that is needed for this is a Kafka consumer that connects to the respective topic(s) and extracts the events. If activity as topic architecture is chosen, this component also has to order the events using the timestamps provided by Kafka. The extraction of events is straightforward due to the standardized serialization format in the event streams. It must be noted that object-centric event logs might require additional event correlation eforts. The extracted event logs can be easily output as XES or OCEL logs or directly used by a (streaming) process mining tool.
Apache Kafka has been used in projects related to various topics such as process mining, digital twins, IoT, and software development. However, these projects merely use Kafka as a middleware to exchange data between components. To the best of the author’s knowledge, no paper shows explicit considerations about the format of the data and the strategy of using Kafka topics.
[
[
Camunda process orchestration platform has already developed own Kafka connectors9
and provides examples of using them for process automation [
Process mining can help organizations gain insights into their processes and find improvement potentials. Data extraction, however, poses significant dificulties as not all systems supporting business processes store the data in a format suitable for process mining. With the advent of event-driven architecture, where event streams are used to communicate between loosely coupled components, these dificulties can be solved by using data architecture suitable for process mining.
In this paper, a system architecture is proposed that uses Kafka event streams both as a platform for communication between systems responsible for the execution of a business process (e.g. BPMS, ERP systems and automated (micro-)services) and as a data storage for online and ofline process mining. To this end, this paper proposes data architecture for Kafka topics, event serialization format, and a process mining component connecting the system to state-of-the-art process mining tools and techniques. In addition, it discusses the possible role of a BPMS in such setting.
Future work involves implementing the proposed architecture as well as evaluating its usefulness in simulated and real-life business processes.