Various life-cycle approaches to Business Process Management (BPM) have a common assumption that a process is incrementally improved in the redesign phase. While this assumption is hardly questioned in BPM research, there is evidence from the eld of AB testing that improvement concepts often do not lead to actual improvements. In this thesis, we propose a methodology named AB-BPM and a set of supporting techniques that facilitate rapid validation of business processes by conducting sequential experiments. We evaluate our methodology and techniques with real-world and synthetic case studies.
Business process improvement ideas often do not lead to actual
improvements.Works on business improvement ideas found that only a third of
the ideas observed had a positive impact [
Contemporary BPM research does not provide techniques and guidelines
on quickly testing and validating the supposed improvements in a fair
manner. There are two major challenges for such an immediate
validation. The rst one is methodological. Classical BPM lifecycle approaches
build on a labour-intensive analysis of the current process, which leads
to the deployment of a redesigned version [
First, we develop a simulation technique that estimates the performance of a new process version using historical data of the old version. Since the results of simulation can be speculative, we propose shadow testing as the next step. Our Shadow testing technique partially executes the new version in production alongside the old version in such a way that the new version does not throttle the old version. Finally, we develop techniques for AB testing for redesigned processes with immediate feedback at runtime. AB testing compares two versions of the deployed process by observing users responses to versions A/B, and determines which one performs better. We propose two algorithms, LTAvgR and ProcessBandit, that dynamically adjust request allocation to the versions during the test based on their performance. We evaluate these techniques with real world and synthetic case studies. 2
Despite the prevalence of live testing and documentation of the successes, contemporary business process management has not embraced this idea. There are three problems with rolling out new process versions in contemporary business process management practices:
The traditional BPM lifecycle encourages replacement of old versions with new versions. The newly redesigned version makes the implicit assumption that this version improves the business process. User interactions are not observed before the replacement because the new version was never executed in production. So, this improvement assumption is not validated before committing to the new version.
Live testing approaches are shown to have addressed such issues in the
context of websites (e.g. [
There is an inherent risk of executing new versions that have not been
validated in production. We henceforth refer to this as risk of exposure.
Careful o ine analysis, as suggested by the BPM lifecycle, can reduce
the possibility of producing bad redesigns but the risk of exposure still
remains because a redesigned process completely replaces the old version.
Alternatively, live testing approaches follow the motto \test fairly, fail
fast", which means that there is minimal upfront analysis before testing
[
To reduce risk of exposure, advanced AB testing techniques for websites can observe metrics like click-through rate and dynamically adjust user allocation during the test. This ensures that the better version receives more tra c over time. However, such approaches cannot be used as-is in BPM because of the complexities in performance indicators. Process instances can have unique identi ers, and each execution can have multiple process performance indicators such as duration and cost. Quality of a process version can be assessed by observing performance indicators of the corresponding process instances. First, the performance indicators may be tied to duration. The delays in which these performance indicators are observed in uences when and how the allocation of users to the versions are adjusted. Second, not all of the required performance indicators may be observable at the same time. For instance, duration and user satisfaction scores may be obtained at di erent times with delays of varying length. Furthermore, indicators like user satisfaction may not be observed at all. Finally, the performance of a process can also be in uenced by factors such as resource constraints, the environment, and market uctuation. To adopt advanced AB testing techniques in BPM, we require monitoring support, mechanisms of evaluating process versions in presence of these complications with performance indicators, and request allocation algorithms for the BPM scenario. 3
This research makes original contributions in the area of business
process improvement, business process simulation, and live testing. In this
research, we have proposed a business process improvement methodology
named AB-BPM [
Our simulation technique takes historical data (event logs) of a process
version, the process model of the new version, and estimates of the
metrics of some activities in the new version as the inputs. It produces event
logs for the new version, which is then used to estimate the performance
of the new version. We construct a Transition Simulation Tree (TST), a
rooted tree data structure that summarizes decisions and metrics
available in an event log of the existing process version.The TST created
from the old version is used in conjunction with a BPMS to derive the
execution log of the new version [
g n it sT e
(Version B) +
(Version A) +
(testVeenrvsiiroonnmBent)
R i s k o f E x p o s u r e The risk of exposure is low in simulation because this technique is performed o ine. However, the results can be speculative because of the implicit assumptions about customer behavior and the explicit assumptions about the execution of some tasks in the new version.
Shadow testing executes the new version in production but hides the
execution from customers. User requests from the production environment
are duplicated and executed in both versions. Execution of the new
version can a ect the performance of the old version during test execution.
Therefore, the risk of exposure is higher than simulation. However, the
new version is executed online and the results are less speculative.
In our approach of shadow testing, we reduce the risk of exposure further
by partially simulating the new version and instantiating the new version
only when the performance overhead of testing is acceptable. As a result,
there is a trade-o between speculativeness and risk of exposure. This
is achieved through a new modular BPMS architecture, an overhead
management algorithm, and a decision tree for partial simulation [
AB testing executes the two versions in production such that each version
serves a portion of customers. This approach has high risk of exposure,
but the results of the tests are fair and reliable. To mitigate the risk of
exposure, we use multi-armed bandit algorithms that dynamically allocate
user requests to the process versions based on their performance. These
algorithms observe performance indicators of each version, and shift the
user requests towards the better performing version in the given context.
E ectively, the best version can be identi ed by the end of the test.
We propose a modular architecture that supports the execution of such
algorithms. We design two algorithms, LtAvgR and ProcessBandit, for
the BPMS case and propose reward designs for these algorithms [
We evaluate this solution with real-world case studies from the domain of building permit approval, banking, and helicopter license approval. We also demonstrate the e cacy of the algorithms with experiments on convergence to optimal solution and scalability using synthetic cases.