The aim of this thesis is to develop and evaluate methods to enhance the quality of predictions for predictive business process monitoring, focusing on the accuracy and stability of predictions. Three different approaches to increase the accuracy and stability of predictions are suggested. Firstly, to improve the accuracy of predictions on minority classes, di erent resampling techniques are applied to data samples in event logs and the accuracy of the predictions from these resampling techniques are compared, and new metric that considers di erent weights of activities is developed. Secondly, the stability of predictions of di erent data distribution-resampling technique pairs is compared. Lastly, the stability of predictions of di erent case types in event logs is compared, and a new performance metric that considers di erent case types and provides balanced predictions is suggested. Besides evaluation using publicly available event logs, a case study is also conducted using a real-life event log from a large hospital in South Korea.
The goal of predictive business process monitoring is to predict various aspects
of interest of business processes, such as the next event in a case [
While techniques proposed in the literature are thoroughly evaluated from
the point of view of performance, e.g., accuracy of f-score of predictions, we argue
that little emphasis has been put on the practical applicability of the prediction
results. Therefore, it remains unclear to what extent predictive monitoring can
improve the business processes in practice. Speci cally, the data mining and
machine learning literature stresses that other dimensions of quality of the results
achieved by a model, besides accuracy, should be considered in order to achieve
a comprehensive evaluation of a predictive model. In particular, the accuracy
and stability of the predictions have not been examined enough in predictive
business process monitoring, while they are deemed important in data mining
and machine learning [
For a clearer understanding, let us consider the following two examples. First,
let us assumes that a manufacturing company needs to detect anomalous traces
in a process as early as possible in order to minimize the cost of defects resulting
from anomalous process execution. Since anomalous traces usually do not
account for the majority of traces, enough data samples for anomalous traces may
not be available, which hinders the capability of a classi er of learning a model
of decent quality for tting the anomalous traces. Assuming that the
anomalous traces account for 10% of traces, and the prediction accuracy measured by
f-score is 0.9 and 0.1 for the normal traces and anomalous traces, respectively,
the average accuracy of this classi er is 0.82, even though the prediction
accuracy of the anomalous traces is only 0.1. To conclude, the gap between the
prediction accuracy of the minority class and the average prediction accuracy is
immense, thereby making the classi er virtually unusable in practice. In the
second example, let us assume that an insurance company wants to predict the next
activity in running processes in order to e ciently handle the ongoing workload.
In this case, predictive monitoring would be useless in practice if predictions
and performance are not stable across a variety of dimensions, such as time, i.e.,
predictions are di erent with every di erent trial, or distribution of case types,
i.e., predictions are very accurate for speci c case types and highly inaccurate
for others. While research has analysed the temporal stability of predictions in
outcome-based predictive monitoring [
In this context, this thesis aims to bridge the gap between academia and practice by proposing techniques for enhancing the accuracy and stability of predictions in predictive business process monitoring. This thesis will focus on the prediction of next activity in a case and process outcomes. As the prediction of next activity is a sequential multi-class classi cation task, while the prediction of the overall outcome is a non-sequential binary classi cation task, preprocessing schemes for these two types of predictions will be di erent. For the data, we will consider publicly available event logs and also real-life event log from a large hospital in South Korea. A case study will be presented using the real-life event log in order to enhance the practical applicability of the results of the thesis.
This thesis was scheduled to be completed in three years, where approximately two years are remaining for now. During the rst year, the literature on accuracy and stability of predictions in predictive business process monitoring, resampling techniques in machine learning, and trace clustering techniques in process mining were thoroughly investigated. In addition, di erent resampling techniques for imbalanced learning were partly implemented. For the second year, the remaining analysis of the rst approach and a case study will be conducted. For the case study, the collection of real-life event log from the hospital will be nished and the evnet log will be preprocessed. Especially in the process of collecting the real-life event log, domain experts in the hospital such as doctors and nurses will be interviewed for the advice of di erent weighting schemes for di erent activities. In addition, probability distributions will be tted to either augmented or reduced data for the second approach. In the last year, stability of the prediction from each "data distribution - resampling technique" pair will be calculated to gure out the optimal conditions for stable predictions. Finally, various trace clustering techniques will be applied and the stability of the prediction from each trace clustering technique will be calculated for the analysis of third approach. After the experiments, the limitation of the paper will be unveiled and concrete scope of the future research will be determined based on the assessed limitations. 2
At the current stage of the research, three existing issues in accuracy and stability of the predictions in predictive business process monitoring have been identi ed. These three issues concerns predictive monitoring use cases that are solved using classi cation techniques, i.e., prediction of next activity in a running trace and prediction of process outcomes. 2.1
The imbalance in class ratio of features in an event log decreases accuracy of
the predictions. Most popular classi cation techniques show poor performance
when they have imbalanced data as input data [
Over-Sampling Over-sampling techniques augment the number of samples in the minority class by sampling from the minority class. Three existing oversampling techniques will be compared.
Under-Sampling Under-sampling techniques delete samples from the majority class. Ten existing undersampling techniques are compared.
In practice, the problem of imbalanced class ratio becomes more evident when the minority class is considered to be a critical task in the operation of the process. In this case, wrong prediction of minority class may bring huge damage to those who run the corresponding process even though the classi er is good at predicting the majority class. In order to account for the importance of predicting the minority class, the penalty is given to the recall of the minority class, thereby modifying the f-score. To better demonstrate the e ectiveness of resampling techniques, a case study will be presented using a real-life event log from a large hospital in Korea.
Di erent data distribution may a ect the stability of the predictions. Once the resampling technique is applied, the data will either be augmented or deleted. In addition, the way in which the data is augmented or deleted varies depending on which resampling technique is applied. In other words, the distribution after the application of resampling techniques can be totally di erent from the distribution of the data where di erent resampling techniques are applied, even though the initial data distribution was identical. Therefore, this thesis compares prediction performance and the stability of di erent "data distribution-resampling technique" pairs as in Fig. 3. In a bimodal distribution where the value of one mode is slightly higher than the other mode, for instance, one oversampling technique may augment the number of minority class by focusing only on increasing the samples that belong to the mode that has higher value, whereas the other oversampling technique may augment the number of minority class by increasing the samples that belong to both modes. In this case, the stability of the classiers of two oversampling techniques is expected to be di erent. Moreover, there may exist speci c types of classi ers that can generate more solid rules for tting the data depending on di erent distribution of the input data. Therefore, it is expected that the best classi er for each resampling technique will be revealed after experiments.
In addition, e ects of di erent sizes of the data modi ed by resampling techniques on accuracy and stability can be an another subtopic of interest in this context. For example, over-sampling techniques may relatively show greater accuracy and stability on huge dataset, but not in a case where the majority class has 10 samples and minority class has only 1 sample. In this case, under-sampling techniques may relatively show greater accuracy and stability where the oversampling technique has to over-sample based on only 1 sample, thereby resulting in a poorer accuracy and stability. Here, we will use case-based strati ed sampling that samples data based on case-level, which means that the distribution of the data is maintained by deleting cases, not by deleting individual events. 2.3
Di erent case types may a ect the stability of the predictions, which means
that the prediction accuracy may uctuate at times depending on di erent case
types. Therefore, this thesis compares prediction performance and the stability
of di erent case types as in Fig. ??. There have been several research regarding
grouping the cases based on features [
It is expected in the rst approach that the prediction accuracy of classi ers with resampling techniques calculated by new metric would be higher than that of a baseline classi er without any resampling techniques. Beyond these results, it is expected in the second approach that speci c distribution of event logs whose samples are either augmented or deleted may show stable prediction accuracy with speci c classi ers. Similarly, in the third approach, it is expected that clusters of cases from some trace clustering techniques may exhibit stable prediction accuracies across every trial. For future works, four improvements can be made. Firstly, further e orts can be made on developing ways to give penalty to the wrong predictions of the minority class. Throughout this paper, it was suggested that f-score should be modi ed by penalizing the recall of the minority class, which means that the penalty is imposed after the classi er is tted and completes the prediction. However, the penalty can be given not only by modifying the f-score but by modifying the way the classi er calculates the error in learning procedure. In case of neural network-based classi ers, cost function can be modi ed, while in tree-based classi ers, cost complexity or entropy can be modi ed. In these cases, conventional performance metrics will be applied without any modi cation. Secondly, better ways to de ne "important" minority class can be developed. This is because in real-life cases, there can be a huge number of minority classes, while some of these cases are not "important" classes but merely minor or deviant classes. Thirdly, other advanced quality metrics of predictions from the literature in machine learning except for the stability would be implemented to improve the quality of predictions for predictive business process monitoring. Lastly, generative models can further be applied in oversampling the minority class since not enough over-sampling techniques are considered in this thesis compared to under-sampling techniques.