-A major challenge in applying process mining on real event data is the presence of noisy or incomplete cases or unusual behaviors. Applying process mining on raw event data leads to wrong conclusions during the discovery of process models, concealing the typical behavior. In this paper, an alternative for filtering event data without the need for extensive preprocessing is presented. The method is based on generated footprint matrices of randomly pruned sub-logs and works in a semi-automated manner. By identifying the most similar matrices to validate the whole log, traces representing unusual behavior can be excluded or highlighted. The tool was implemented with Python 3, NumPy and Pandas and is publicly available on GitHub. We evaluated our tool using benchmark data-sets and compared it to human filtering and discovery results. Index Terms-Data mining; Process mining; Preprocessing II. MATERIAL III. METHODOLOGY
Process mining represents an essential and evolving part
in the field of data analysis. It provides ”fact-based insights
and supports process improvements of business processes” [
This proposed tool and its algorithm represent an approach
for an automated cleaning task, which can find the most
frequent control flows in an event log. The whole technique
is based on the concept of footprint matrices, which were
introduced with the alpha algorithm. [
This method can clean event logs from unexpected behavior (i.e., rare sequences of events), it additionally allows inverting the result and highlight cases that represent rare or infrequent behavior.
A combination of different software tools and libraries were
used to create the Process Pruner tool. The actual program was
created with Python 3.6 using the NumPy1 and the Pandas2
library. To visualize the results, ProM version 6.8 and Disco
were used. Especially the included Inductive visual Miner
[
Before the actual pruning algorithm starts grouping the traces as seen in Fig. 1, an additional preprocessing can be applied. This stop targets the end events of every trace.
If the user provides a list of valid end events, then this
information is used as a preprocessing filter before the actual
algorithm starts. Each matrix is generated out of K amount
of traces (default 50) which are generated by grouping all
traces of the log by their identifier. After shuffling the resulting
identifiers, K of them are chosen to constitute a sub-log. This
process is repeated N times till N amount of footprint matrices
are generated. The generation follows the rules described
within the alpha algorithm [
similarity = count(similarEntries) count(entries) (1)
Fig. 1. All cases of the log are group depending on their case id to get the whole trace. For each of the N generated footprint matrices, K traces are randomly selected (i.e. each trace is only used for the generation of exactly one footprint matrix). After calculating the similarity of each matrix to all the others, the most uniform one is used to validate the event log. The final log only contains the traces which could be replayed by the selected matrix.
This method works under the assumption that footprint
matrices contain pairs of directly following events. Based on
pairs, a full footprint matrix represent a sequence of pairs
which express a possibility of the underlying event log [
This tool with its algorithm can be used in a pipeline, where first filtering should happen to gain a better overview. This tool contains an interactive command-line interface, which allows to define the source CSV-file, the output-files and setting the trace and event identifier. Additionally, it allows filtering before the primary analysis by an end event criteria, which can be skipped. The output contains two files – one containing the matching traces and one the non-matching traces. The tool is also publicly available on GitHub3 with a demo video4.
The proposed method is tested with two publicly available benchmark datasets. The tool needs a CSV-file for input. eXtensible Event Stream-files (XES) must be converted to a CSV-file, e.g. via ProM or Disco. For the evaluation, the default value 50 was used for the variable K.
a) Road Traffic Fine Management Process [
By applying a variant filter only keeping the traces that
appeared more than once, 131 different variants of traces
remained. The original event log contains lots of
irregularities, i.e. there are many different process variants [
The results between the different preprocessing can be seen
in the Tab. I. The Process Pruner almost achieved the filter
quality like manual filtering without any insights into the
dataset.
b) Artificial Event Log - Hospital example [
As seen in the Tab. IV, there are more than one ways of how
to start the process. This should not be possible, because the
log is generated. As described in [
This example clearly shows the potential of this automated preprocessing algorithm. Without any parameters set it was possible to filter incomplete traces and reduce the log in a way that the core process becomes visible. The Tab. II shows metric information for this event log. The original metric is therefore set against filtering by 80 percent variant with Disco and filtering with the Process Pruner. The table shows that the variant already reduced the most varying traces. The Process Pruner was able to remove more and was able to keep the primary process which is shown in the Tab. V.
To prove that this is also possible with a noisy dataset, the used dataset, therefore, provides the base event log with 30 percent noise. The Tab. III contains the metrics of the original dataset and the variant filtering and the filtering with the Process Pruner. Our tool was able to find the primary process and removed about 50 percent of variants.
As seen in Tab. VI it is also possible to reduce noise within a log and get an approximation of the core process. Even if there are some differences compared to the previous example, it was possible to reconstruct parts of the core process. This includes the information that every trace in this context has to start with the ”Triage” event and the ”Organize Ambulance” event could only be placed after all the other ones. It is also clear that ”X-Ray” and ”Check” are compulsory events.
This example can be seen as a demo of how the presented algorithm can be used. If the user has to manually filter the invalid traces out of a dataset with 30 percent wrong events, it is for sure not as easy as to use this algorithm which could be run more than one time to get more and more stable results. Even with noisy data, it is possible to get an insight into how the real world process could be.
Compared to a manually done preprocessing, the presented workflow clearly shows its strength in the field of filtering incomplete traces and noisy event logs. Especially for event logs with a high amount of events, it is not trivial to get an insight which traces are valid and which ones are not. Therefore it is time and cost saving to use preprocessing algorithms like the one presented in this paper to get stable results which deliver a good approximation of the real world processes.
Even data logs with a high amount of noise could be used to discover the core processes hidden in the log. This also delivers the possibility to isolate the noise producing parts of the real world by merely applying the algorithm in the way that not the most similar footprint matrices were used, instead of the more unique ones, which are representative for the noise and unusual behavior within the log. The combined information of the core process and the error producing and unique process instances deliver very detailed insight into the real world processes which could otherwise only be possible by investing much time by manually extracting the information out of the event log.
This tool was designed and implemented by students in the Process Mining class of the Data Science and Engineering Master’s program at the University of Applied Sciences Upper Austria in Hagenberg. It will be used in future lectures to compare the students’ manual data clearing efforts with an automated approach and to analyze the benefits and weaknesses of automated log filtering. With the next release, we plan to support XES-files as sources for this tool and plan to integrate it into PM4PY.
As the importance of process mining as well as the amount of collected log data rises, new preprocessing approaches are needed. We showed that our automatic preprocessing tool could help to improve the quality of logs by filtering incomplete or rare traces. Furthermore, we showed that the Process Pruner could be used to detect rare behavior, which might be interesting in scenarios where the rare traces have a significant impact on the process. More research has to be done in this area to make the preprocessing more accurate in terms of finding the most exciting cases. > # < < < # # # > # < < < # # # k k k k k k k k # > k k k k k # # > k k k k k < # > k # k k > < # > k # k < # # # > k # k < > # > k k k # < k > # > k k # < # # # > k k # < > #
X-Ray
X-Ray
> # k < k k k # # < # # # # # #
X-Ray
Organize Ambulance k k k k > # < # # # k > > # < # # # k > > # < # < k k < k > # < # # k # # > # < # # k < < > k < # # k > < # > # # # # # # # > # # # > # # # > #