II. METHODOLOGY
Process mining is a family of process analysis techniques
that enables the discovery of process models from an event log
(process discovery), the ability to check conformance between
the actual and the assumed process (conformance checking),
and other process-related analyses to enhance the process
(process enhancement) [
Although conformance checking has great potential in
detecting process deviations, challenges exist that hinder it from
full adoption in auditing practice. First, because a normative
process model is an idealistic and simplified representation
of a process, performing a conformance check results in a
large set of process deviations. This makes it impossible for
the auditor to check each deviation one-by-one and therefore
forces the auditor to take samples instead of auditing the
complete set of transactions [
In this doctoral research, we aim to cope with the challenges associated with conformance checking in auditing by focusing on the classification of process deviations. For this purpose, we develop an active learning framework that combines machine learning techniques with domain expertise. The framework provides auditors with a practical approach to audit the complete set of transactions instead of a sample. The idea is to use conformance checking output (deviations) and combine this with domain expertise in an iterative process.
This extended abstract is structured as follows. In Section II, we describe our research methodology and walk through the different steps of our framework. An overview of related work is given in Section III. Finally, we conclude our research in Section IV.
The doctoral thesis will consist of designing and testing a deviation classification framework. In this section, we present the current state of the framework and provide some information on how we will validate our study.
In our specific context, we are researching how we can make conformance checking feasible in auditing practice. To this end, we propose an active learning framework for the classification of process deviations. The proposed framework consists of the six following steps:
1) Label a small sample of the unlabeled set of deviating cases: The framework’s goal is to label a set of deviating cases that were detected using conformance checking. The possible output labels for each deviating case are either anomaly or exception. An anomaly is a case that consists of at least one anomalous process deviation. An exception is a case that consists of only exceptional process deviations.
The framework starts with taking a small sample of a set of deviating cases that an auditor has to label as anomaly or exception. The labelled small sample is the input for the second step of the framework.
2) Mine rules from the labelled set: In the second step,
we use the small labelled sample to mine rules. We want to
discover two types of rules: control flow relations and other
data relations. Control flow relations refer to relations between
the order of the activities in a case. To discover such relations,
we are planning to use a declarative constraint miner [
3) Transform rules to labeling functions: The mined rules
from step 2 are in this step transformed into so-called labeling
functions. labeling functions are a construction of rules used
in the tool Snorkel. Snorkel is a system that implements
weak supervision to train machine learning models without
the need for labelled data. Instead, Snorkel uses labeling
functions that encode domain knowledge in the form of rules
[
Snorkel because it can indirectly use domain knowledge to guide the labeling of individual cases.
4) Combine labeling functions in a machine learning model
and label the remaining unlabeled cases: In this step, we
combine the labeling functions of Snorkel in a machine
learning model. Snorkel has a built-in generative model, called the
LabelModel. The LabelModel weights and calculates labeling
function accuracy to assign a final weighted label to each case
in the data [
5) Calculate the accuracy: In the last step of the framework, we calculate the model accuracy by using a labelled validation set. If the accuracy is below a threshold (that is beforehand defined by the auditor), then we re-iterate over the framework by taking an additional sample of the unlabeled data. If the accuracy is above the predefined threshold, then the framework stops here, and the auditor ends up with the set of labelled cases. The auditor can now focus the audit on anomalous cases.
The results of this doctoral study will be validated in two steps. First, we artificially create event logs and process models on which we apply our framework. Our approach will be validated based on the experimental results. We plan to look at the model accuracy, the impact of high or low-quality rules on model performance, the impact of the number of rules on model performance, and the number of iterations over the framework.
Second, after succeeding in the artificial setup, we apply the framework in a real-life setting. For this purpose, we are working with a Big Four auditing firm. In this stage, we will again look at model performance, the impact of the quality and quantity of given rules, and the number of iterations over the framework.
Since data is widely available and technology is becoming
better, auditing is forced to change [
Although previous research proposes frameworks to cope
with alarm floods in continuous auditing [
In this doctoral research, we want to enable full population tests in auditing by making conformance checking more feasible in practice. We propose an active learning framework that combines conformance checking output with domain knowledge with the goal to label deviating cases as anomaly or exception. The research follows a two-step approach. First, we set up some experiments on artificially generated event logs and process models. After that, we test the framework on a real audit environment of a Big Four auditing firm.