Information systems log data during the execution of business processes in so called “event logs”. Process mining aims to improve business processes by extracting knowledge from event logs. Currently, the de-facto standard for storing and managing event data, XES, is tailored towards sequential access of this data. Handling more and more data in process mining applications is an important challenge and there is a need for standardized ways of storing and processing event data in the large. In this paper, we first discuss several solutions to address the “big data” problem in process mining. We present a new framework for dealing with large event logs using a relational data model which is backwards compatible with XES. This framework, called Relational XES, provides buffered, random access to events resulting in a reduction of memory usage and we present experiments with existing process mining applications to show how this framework trades memory for CPU time.
Over the last few years, the amount of historical execution data stored by large-scale information systems has grown exponentially. This data is typically stored for analysis purposes, for example to enable auditing, process improvement or process mining. The evolution of information system into such large-scale systems increases the need for business process analysis techniques to also handle data at such a large scale.
Most process mining techniques that exist today focus on the analysis of so-called event logs with the purpose to discover, monitor and improve business processes. Traditionally, process mining techniques assume that it is possible to sequentially record events into traces such that each event refers to an activity (i.e., a well-defined step in the process) and is related to a particular case (i.e., a process instance). Event logs may store additional information such as the resource (i.e., person or device) executing or initiating an activity, the timestamp of an event, or data elements recorded with an event (e.g., the size of an order).
This classic sequential view on a log data has a few important downsides. First, each event is assumed to be relevant for only one trace in the context of one process. In real life this is not necessarily the case, i.e. events may be relevant in the context of many different traces belonging to different processes. Letting an event appear in many traces requires a significant amount of duplication. Second, the tree structure of an event log is great for sequential access to the log, but not suitable for random access, while most techniques actually use a random access paradigm to access events in the log. Third, the current rise of decomposition and distribution based techniques for process mining requires easy filtering of event logs both vertically, i.e. distributing cases over different logs and horizontally, i.e. distributing events over different logs.
In this paper, we present an architecture that addresses the issues above while it maintains backwards compatibility with existing process mining techniques. Our architecture uses a database to store the event log, allowing for events to appear in multiple traces, for events to be accessed in a random-access fashion and for efficient filtering.
The remainder of this paper is structured as follows. In Section 2 we discuss related work. Then, in Section 3, we present our framework for storing and managing event data. In Section 4 we compare our framework with the de-facto standard for managing event data and we conclude the paper in Section 5. 2
Process mining is a research discipline that provides techniques to discover, monitor
and improve processes based on event data. It is beyond the scope of this paper to give
a full introduction to process mining, but we refer to [
Most process mining techniques today have been assuming that event data is
available in the form of sequential traces of ordered events that are typically ordered in time.
More recently however, process mining researchers have come to realize that events are
typically related to multiple processes being executed in parallel while synchronizing
on some activities [
Since the start of research on process mining, attempts have been made on
standardizing the way in which event data is to be stored. This has lead to a number of
semi-standards, such as MXML [
OpenXES is a reference implementation for dealing with event data in the XES
format. This reference implementation consists of a number of interfaces as well as a
number of concrete implementations. These interfaces allow for both sequential and
random, read and write access to event data. Over the years, this implementation has
proven to be very successful in the open source process mining framework ProM [
It is not the first time that databases are used to store event logs. XESAme [
XES is an open standard for storing and managing event data. For the purpose of storing
event data, a standardized, extensible storage format was developed, of which the
definition is shown in Figure 1. In XES, each event, trace and log is annotated with typed or
untyped attributes which are given semantics through so-called extensions. For
example, the activity an event refers to is stored as a literal attribute with key “concept:name”.
This key is defined in the standard “Concept Extension” [
One of the fundamental assumptions of XES is that each event belongs to exactly
one trace and occurred in the context of exactly one log as shown in Figure 1. In practice
however, this is often not the case [
Therefore, in this paper, we present the RXES framework that lifts this assumption. Instead of considering an event to have occurred in the context of a particular trace, we consider a trace to be a collection of events and a log to be a collection of such traces, but events may occur in many different traces and traces may appear in multiple logs. This allows for backwards compatibility with traditional mining techniques that rely on the “single trace” view, while also allowing for more advanced techniques to consider multiple views at once without the need for duplicating these events.
In Figure 2 we show an ER diagram for the RXES framework. The framework uses a scheme to store events in a database that is largely based on the UML class diagram used for XES but separates the contents of events and traces from their appearance in traces and logs respectively, thus requiring significantly less data duplication. In the remainder of this section, we discuss the main differences between XES and RXES. 3.1
In RXES, logs, traces and events are represented by tables with ids. The actual state of an XTEvent is dependent on values of its attributes (accessed through the XAttributable class). Similarly, an XTrace is nothing more than an ordered list of events (represented by the composition) which carries state through its attributes and can only exist in the context of a log. Therefore, there is no need to have content in the tables representing traces and events in RXES.
Events occurring in the context of a particular trace are represented by the trace_has_event table which identifies occurrences of events rather than the events themselves and similarly trace occurrences are represented by the log_has_trace table. As events can appear in multiple traces at different locations and traces can appear in multiple logs at different locations, we keep an order number indicated by sequence that is specific to the occurrence of an event in a trace or a trace in a log.
An event appearing in multiple traces can in RXES be represented by a single entry in the event table, but the schema is flexible enough to allow for duplication of the event for each occurrence if desired. 3.2
In XES, attributes are represented by a single class XAttribute. Each attribute is
composed of a single key and is typed through one of its subclasses (Boolean,
ConFig. 1: UML class diagram of the XES standard [
As shown in Figure 1, event logs may contain a number of global attributes. The notion of global events refers to the idea that a log can specify that a particular attribute is present on all traces (or events) contained within it, i.e. the attribute is defined to be globally present. This allows for process mining techniques to verify whether a log satisfies certain input conditions, such as the presence of timestamps. A global attribute also specifies a default value which can be used for example when adding new traces or events to a log or in case an attribute declared to be global is nonetheless not present. In RXES, we use two binary attribute in the log_has_attribute table to indicate if an attribute is global. To cover the concept of global attribute it has been enforced by the schema that there should not be two global attributes in one log with the same key but a different value. 3.3
Another feature of XES is the availability of so-called classifiers and extensions which provide semantics to events in a log. A classifier in XES is composed of a collection of attribute keys. If a classifier is included in a log, it provides insights into the way individual events should be translated into business activities (or event classes in XES terms). The idea is that such classifiers provide a starting point for process mining techniques to reason on the correct level of abstraction. In practice, classifiers are rarely contained in a log and are often added by the process mining technique. Therefore, in our database, we represent the attribute keys of classifiers simply by text. As a general rule, a classifier should only refer to keys of event global attributes, but this is not enforced by XES nor by RXES.
The last concept of XES that is supported by RXES are the so-called extensions.
The extension mechanism allows users to give semantics to attributes. These extensions
specify specific keys for attributes which have to be interpreted in a standardized way.
For example, the concept extension defined by [
The attribute table uses auto-generated IDs attached to each attribute to connect attributes of the same type using SQL queries. By using an id as a primary key rather than other fields, we allow for recognition of identical attributes which is quite useful for decomposition, filtering and importing. When encountering an attribute that exists in the whole log, we may add a reference to the existing id in the database using an inmemory cache of existing attributes. However, if an attribute is not in cache, we add the attribute using a fresh id. The consolidation of identical attributes can be done offline. 4
RelationalXES provides a full implementation of all OpenXES interfaces using the database as a backend. As a result the framework is fully backward compatible and provides a number of benefits over OpenXES. First, keeping events in the database and only retrieving minimal amounts of data per each request, reduces the memory usage of process mining techniques significantly. Second, through SQL-based filtering and database views, decomposition algorithms no longer require event data to be duplicated in memory, and third, events can exist outside of the context of a trace or a log, allowing RXES to store event data from streams and to utilize trace identification techniques.
RXES has been implemented in Java ,and for the experiments in this paper, MySQL has been used as a database system. However, the design of the application allows for any relational database systems. The framework keeps only the data items that are directly related to the log entity plus list of ids of traces in the memory and later loads the actual data if it is necessary, i.e. if it is requested by a process mining technique. RXES is implemented as a package in the ProM framework and is available as an opensource implementation. It can be downloaded from http://www.processmining.org/.
Currently OpenXES has multiple implementations included in ProM. In this section, we show the difference between our new technique and existing implementations by evaluating memory and CPU usage of processing different sizes of logs.
To investigate memory and CPU usage we used 30 sample event-logs. For that,
we used a public, real-life dataset [
Clearly, the default OpenXES implementation (using Java’s Collection Framework) has the highest memory allocation. The existing MapDB implementation (in package XESLite) drops the usage by 90%, while our technique uses 90% less than that. For RXES, these results include the memory needed by the Database Management System.
Figures 3(c) and 3(d) show a comparison for the time needed to access all events in the event logs. To evaluate time, a program that requests all elements of the event log has been executed 10 times and an average time is computed. The amount of memory given to the Java virtual machine is sufficient to meet the demands of the Default OpenXES implementation. Clearly, there is a direct relation between memory usage and speed of access. As Figure 3(c) shows, the default implementation is the fastest method simply because the event log is loaded into memory completely. The figures further show that (a) Memory use vs. number of events (b) Memory use vs. number of attributes b M s 1;500 1;000 500
RXES is slower than MapDB which uses an in-memory database, which is mostly because for this experiment we used minimum size of the trace buffer which only keeps one trace at a time. Hence, the execution time is result of performing many SQL queries over a TCP/IP connection to the DBMS.
In general, if the size of the event log is small compared to the available memory, MapDBDisk keeps a good balance between memory and time usage. However, compared to RXES, MapDB doesn’t allow for distribution and replication, while it is vital in cases of using decomposition and parallelism to handle the big data.
The ability use of RXES to process large event logs with readily available process mining techniques is essential for the adoption of our framework in practice. However, our framework is capable of performing tasks directly on the database, such as event log filtering. 5
In this paper, we presented RelationalXES. This framework is a generalization of the de-facto standard XES for storing and managing event data in process mining. Where all existing implementations of XES uses the strict notion of containment for events in traces and traces in logs, our framework is much more flexible and allows for events to appear in more than one trace and for traces to appear in more than one log. That makes it possible to have different views on the same event log. Furthermore, the database schema used in RXES allows for a significant reduction of duplication by storing frequently occurring attributes only once rather than repeating them for every occurrence.
In the paper, we presented the framework in detail and we discuss the underlying database schema. Furthermore, we show the reduction of memory use by process mining techniques using RXES.