Predictive process analytics focuses on predicting the future states of running instances of a business process using machine learning and AI-based techniques. While advanced machine learning techniques have been used to increase accuracy of predictions, the resulting predictive models lack transparency. Current explainability methods, such as LIME and SHAP, can be used to interpret black box models. However, it is unclear how t for purpose these methods are in explaining process predictive models, which use complex, multi-dimensional event logs, often alongside various types of context-related data. However, given the vast array of explainable methods available, and the di erences in the mechanism used to provide the explanations and the content of the explanation, this evaluation becomes complex and no standard method or framework of evaluation currently exists. The proposed project, therefore, aims to address methods to evaluate and improve AI and machine learning transparency in the eld of predictive process analytics, with particular emphasis on creating a standardised evaluation framework and approach for event log data.
Business Process Management (BPM) methods are increasing in technical complexity and sophistication. An emerging BPM tool is predictive process analytics (PPA), which, at runtime, attempts to predict some future state of a process using machine learning models [11]. Modern data analytics techniques and increased availability of machine-generated process data has enabled PPA, which applies predictive analytics to business processes, providing a powerful tool that can be used to support better decision-making in organisations, optimisation or other process management activities [15].
However, the opaque nature of some prediction systems are cause for concern.
While more complex and sophisticated prediction algorithms often produce more
accurate predictive models, these models are also less transparent { an issue
that could a ect an organisation's transparency, ethical conduct, accountability
and liability, and raise potential issues with the safety of and fairness towards
stakeholders [
The proposed project, therefore, will address methods to improve AI and machine learning transparency for predictive process analytics, provide a framework for evaluating these explainable methods and attempt to provide a set of guidelines or recommendations for PPA explainability. There will be a focus on creating explanations to empower decision-making, with emphasis on explanation understandability and comprehensibility. This paper is structured as follows. Section 2 introduces PPA, explainable PPA and XAI in more detail. The research gaps to be explored by the proposed project, and the approach for the proposed research are explained in section 3, and progress achieved to date is presented in section 5. Planned future work is highlighted in section 6, and section 7 concludes this paper. 2
Process mining forms the backbone of PPA, where event data is extracted from historical event logs, and used by a prediction model to predict some future state of running process executions (known as the prediction target), such as how a running process will end, time to the end of the process, or future sequences of activities [11]. Most techniques for PPA require data processing, followed by learning and prediction, though there may be some variations depending on the data and learning algorithms used. To improve prediction accuracy, event log data can be supplemented by contextual information that might prove useful in making the nal prediction, such as case documentation [11].
A two-phase approach is generally applied to PPA [20, 21], which begins with an o ine processing and learning phase where the predictor is created and trained, followed by deployment of the predictor at runtime (see gure 1). The rst phase begins with the processing and combining of historical event log data and contextual data to extract relevant process information. Generally, the relevant information, including activities that have been completed, attributes associated with each activity, and any contextual information, are grouped together by the case for which that activity occurred, creating a log of pre xes that describe a single run of the process { a single process instance. These prexes then converted into features more appropriate to train a predictive model, and used to create a predictive model. This processing and conversion is also necessary in the second phase, when runtime data is used to predict some future state of a running process instance.
One complexity in predictive process analytics comes from the data used. Event log data alone is highly complex, containing various attributes for each activity in a case, including static attributes that do not change throughout a process instance, as well as dynamic attributes that often do change throughout the process instance, both of which must be encoded, though the degree to which the temporal information is preserved during encoding often varies. If combined with context data, which may or may not be temporal in nature, this encoded dataset becomes even more complex. Given this dataset complexity, and the inherently opaque nature of the machine learning algorithms that are most e ective in creating accurate process predictions [18], it becomes hard to understand why a predictive model may have returned any given prediction. Furthermore, this lack of clarity leads to poorer understanding of model trustworthiness and impairs a user's ability to make informed decisions based solely on a predictive model's output [15].
As such, explainable predictive process analytics has emerged as an attempt
to increase the transparency of these process predictive models. State-of-the-art
explainable predictive process analytics have generally attempted to apply
existing explainable methods to PPA. Interpretability in machine learning is generally
broken down into two categories: interpretable prediction models and post-hoc
interpretation. Interpretable prediction models are those that are generated in
such a way as to be immediately interpretable by a human [
While there have been attempts to apply or create XAI methods to interpret
and explain process predictions, the tness of these methods to explain PPA is
unknown, and there are few frameworks available to evaluate this tness. There
are a number of ways to evaluate explanations, many of which depend on the
purpose of the explanation and the explainee [13]. Although a decontextualised
evaluation will not necessarily provide an indication of explanation usefulness
or user satisfaction [
Design Science Research (DSR) will be used to guide the methods used in the
proposed project. Hevner et al. [
The work will be conducted in three phases: { Phase One will be comprised of the Problem Identi cation and Motivation and Objective De nition stages of the DSR methodology, wherein the interpretability and explanation needs of PPA and relevant explanation methods will be explored (RQ1) and the characteristics required for PPA explanations and ways to determine the tness of explainable methods for PPA will be de ned (RQ2). { Phase Two will be an iteration the Design and Development, Demonstration and Evaluation stages of the DSR methodology that attempts to answer RQ3.1. In this phase, standard approaches and methods for evaluating explanations of tabular data will be developed, assessed and re ned. { Phase Three will be another iteration of the Design and Development, Demonstration and Evaluation stages, this time with the aim of answering RQ 3.2. In this phase, the standard approaches and methods created during Phase Two will be adapted for event logs, assessed and further re ned as necessary.
Results of the research will be communicated throughout all three phases. It is expected that the following outcomes will be achieved as a result of this project: 1. Proposal of evaluation criteria for evaluating explanations and explanation methods for process predictions; 2. Proposal of a standard approach to evaluating explanations for tabular data; 3. Proposal of speci c evaluation methods for classes of explainable methods used to explain process predictive models that are generalisable within those classes; and 4. Functionally-grounded evaluation of several existing explainable methods to determine their tness for explaining PPA;
A three-level system of evaluation that considers context to di ering levels is
proposed in [
This PhD will primarily focus on the rst level of functionally-grounded evaluation. Functionally-grounded evaluation measures include those that do not require human input, but still allow some objective judgement of the explanation to be made. A common, and important, example of such a measure would be computational e ciency [13]. This is particularly important in PPA, where the earliness of interventions in a process instance a ects ability to change undesirable outcomes. Another key requirement for explanations is consistency, also known as explanation stability . This is de ned as the level of similarity between explanations for similar or identical instances [22], and is vital in ensuring trust in the explanation and the predictive model, and ensuring actionability. Also vital is the accuracy of the explanation with respect to the original predictive model, known as explanation delity . This is generally de ned as how faithful the explanation is to the black box { that is, how accurately the external mechanism of the explainable method mimics the black box.
An initial functionally-grounded evaluation was conducted with three event
log datasets and two explainable methods for the outcome prediction problem,
where the nal outcome of any given case was being predicted. The two
explainable methods chosen for initial testing were Local Interpretable Model{Agnostic
Explanations (LIME) and Shapley Additive Explanations (SHAP). These two
methods are currently the most prevalent explainable methods in PPA literature
(for example, used in [
A key challenge for this initial evaluation was, and still is, the lack of
standardised or generalisable methods that can be applied for evaluation and
comparison of the two explainable methods, particularly explanation stability and
delity. Given their relative popularity in the eld of XAI, it was assumed that
existing evaluation methods would exist for LIME and SHAP that can be adapted
for event log data, and later for other explainable methods. Although event
log data are complex, and include a time-series component that sets a
particular sequence to the features present within the data, event logs are similar in
construction to tabular data. Moreover, when processed for machine learning
algorithms, this data is often formatted in such a way that temporal
information is only partially preserved, and the nal input to the model is identical in
format to that of standard tabular data. As such, the initial plans for testing
relied on using existing evaluation methods for LIME and SHAP, and adapting
these methods to consider the temporal aspect if necessary. However, existing
evaluation methods and metrics are often speci c to a particular explainable
method [22] or data types other than tabular, such as images or text [
In order to develop such methods, existing approaches were adapted. In particular, evaluation methods and evaluation metrics used to assess the stability of feature selection algorithms are being applied to test the stability of LIME and SHAP. Much as LIME and SHAP do, feature selection algorithms provide a ranking or subset of the most useful features to use in evaluation, and also often o er some kind of "feature weight" to indicate the importance of each feature, and the stability of all three may be assessed [12,14]. As such, it was decided that explanation metrics used to assess the stability of feature selection algorithms can be adapted to evaluate the stability of LIME and SHAP. In particular, two types of stability are being assessed for feature attribution explainable methods used: stability by subset, which assesses the stability of the subset of the most important features; and stability by weight, which assesses the stability of the weight assigned to each feature as part of the explanation.
The evaluation method for explanation delity has, thus far, proven to be
more complex to develop. The initial approach mimicked a commonly-used
ablation approach for assessing the delity of image and text data, wherein the
features determined to be relevant by the explanation are removed and the change
in the prediction probability for the original prediction is used to determine the
correctness of the explanation [
There are a number of further activities that need to be undertaken to fully answer the research questions outlined. Currently, evaluations have been conducted with LIME and SHAP, which are feature attribution methods { local methods that explain the e ects of di erent components for a single prediction [10]. These two methods alone are not representative of the vast array of explainable methods currently available, and as such approaches that can be used to assess other classes of explainable methods must also be considered. Of particular interest are explainable methods speci c to time series data.
A key challenge here will be to design explainable method evaluation methods in order to ensure comparability between explainable methods of di erent classes. For example, one type of stability currently measure is the stability of weights assigned to each feature. If techniques other than feature attribution are to be evaluated, this measure will no longer be relevant, but other types of stability may become more relevant, such as the stability of predicates in rule-based explanation. However, the results produced by the evaluations must still be comparable while being suitable for the relevant explanation methods, in order to enable the ability to compare and choose between explainable methods given a predictive problem. The ultimate goals of this project will be to create a functionally-grounded evaluation approach to assess the suitability of any given explainable method in the relevant classes of explainable methods for time-series based prediction problems, such as PPA.
Furthermore, the evaluation methods and approaches that will result from this PhD will also also require validation and evaluation. A viable way of doing so may be to use simpler, inherently interpretable and transparent machine learning models when developing the evaluation methods, and con rm the validity of the methods against a transparent model. Existing methods and measures from other, similar elds, are also being adapted as necessary, as is the case with the adaptation of stability measures and methods from the eld of feature selection.
Post-hoc explainable methods are gaining popularity as a means of improving the transparency of process predictive models. However, the tness of these methods for predictive process analytics is extremely unclear. Given that no standard approaches for evaluation of explainable methods exist, particularly for process prediction explanations, the project outlined in this document will attempt to propose evaluation criteria for evaluation explanations and explainable methods for process predictions; create standardised approaches for evaluating explainable methods for process predictions; and propose evaluation methods for classes of relevant explainable methods with regards to explaining process predictions. Acknowledgements I would like to thank my supervisors Dr Chun Ouyang, Dr Catarina Moreira and Dr Renuka Sindhgatta for their ongoing support and guidance.