Past research revealed issues with arti cial event data used for comparative analysis of process mining algorithms. The aim of this research is to design, implement and validate a framework for producing arti cial event logs which should increase discriminatory power of arti cial event logs when evaluating process discovery techniques. { What model characteristics can we identify which in uence the generated data? { What is the impact of model language bias on the generated data? { Which non-model characteristics exist which in uence the generated data? { What is a proper methodology for generating arti cial data for comparative analysis? { Which tools exist for generating arti cial data and to what extent are they su cient?
Literature on the comparative analysis of process discovery techniques has revealed some problems with arti cial data. The data lacked discriminatory power. We argue that such problems arose due to the absence of a proper framework to generate arti cial data. This leads to our main research question: how can we generate arti cial event logs with su cient discriminatory power for a comparative evaluation of process discovery algorithms? To provide an answer to this question several other questions need to be answered:
This work focusses on arti cial data used for the comparison of di erent process
discovery techniques, more speci cally the comparison of control- ow techniques.
In past research on process mining many researchers used arti cial data for the
development of and the veri cation of new algorithms (e.g. [
In a recent study De Weerdt et al. compare several process discovery
techniques on both arti cial and real data [
A lot of process discovery techniques have been developed in the last decade.
Since the rst algorithms, process discovery has matured remarkably. However,
it's still not clear which algorithm will perform best in a certain situation. This
has led to an increasing importance of the research on comparing di erent
process discovery techniques [
The comparison of process discovery techniques can be based on both arti cial and real data. Real data, however, are at a disadvantage when performing such a comparative analysis.
Two disadvantages stem from the nature of algorithm comparison and evaluation. Typically research is performed on a sample of event logs, but conclusions are preferably generalizable to other event logs. To achieve reliable conclusions, statistics require su cient observations and samples which are representative for the considered population. Real data, however, have limited availability and are typically convenience samples, rather than random samples.
Another disadvantage of real data is concerned with identifying causal relationships between process or event log characteristics on the one hand and algorithms performance on the other. This kind of research requires experimental data and not observational data (real data).
In contrast, these disadvantages are not present when using arti cial data in
comparative analysis of process discovery techniques if a proper methodology is
used to generate the arti cial data. Such a methodology should focus on creating
arti cial data with su cient discriminatory power to overcome the problems
encountered in past research (e.g. [
Firstly, a structured literature review is performed to get insight into generating arti cial data and algorithm comparison. The primary sources used to perform this review are: literature in the domain of process mining and literature from other domains on (generating) synthetic data.
Secondly, the general methodology is built and implemented in a tool to support this new methodology.
Finally the implemented methodology is tested and validated by repeating experiments done in past research on comparing process discovery techniques.
One important limitation of this methodology will be its scope which is limited to generating arti cial data for analysing control- ow discovery techniques. Also the reader should be aware that such a general methodology for arti cial data will not replace the need for real (test) data. Real logs continue to be necessary for making arti cial event logs more realistic and as a nal review for process discovery techniques.
The literature review of articles on the evaluation of process discovery techniques
based on arti cial data (i.a. [
To address this issue a preliminary framework is distilled from the literature review which focusses on the crucial aspect of randomization. The methodology can be divided into two stages: the generation of an arti cial process model and the generation of event logs from this model. Both stages allow the researcher to de ne the characteristics of the population and produce a representative sample (see table 1).
The rst step is to de ne a population of process models, from which arti cial models are sampled randomly and automatically. In past research this crucial step in generating arti cial event logs was never made explicit in a general method or guideline. Mostly processes were drawn manually in an ad-hoc manner without explicitly de ning the population they were drawn from. However, it is important that the researcher has insight into the process model population and can in uence the properties of that population. Therefore, ranges for the list of controllable properties (see step 1 in table 1) must be set to de ne the population. Next, values within these ranges are selected randomly and automatically to de ne a single process model.
The second step concerns the generation of event logs for each process model de ned in the previous stage. Again, the researcher must set ranges for several event log properties, from which exact values are sampled randomly to generate event logs. The parameters which can be set are shown in step 2 in table 1.
Di erent tools already exist which can help to automatically generate arti cial event logs. We evaluated two tools considered most appropriate to support the 1. Model Generation 2. Log Generation
Number of activity types Choice structural patterns Choice nested structural patterns Number of generated process instances Required completeness Noise
Imbalance of execution properties
preliminary framework: the PLG tool [
Although the framework as presented in table 1 is still preliminary, it was used in a rst case study to assess if it was a step towards arti cial data with more discriminatory power.
For this case study we repeat part of the experiment of De Weerdt et al. [
We hypothesize that our methodology can produce arti cial data with more
discriminatory power. Therefore we repeat part of the experiment of De Weerdt
et al. [
We applied the preliminary methodology to generate 35 arti cial event logs
out of two random populations using the PLG tool (with all its limitations). Then
four process discovery algorithms were evaluated in two conformance dimensions,
tness and precision, using the method described in [
The results in the tness dimension show that the performance of the tested
algorithms re ect better the results for tness on real data in [
From these ndings can be concluded that the preliminary methodology is only a rst step in the direction of increasing the discriminatory power of arti cial event logs.