=Paper=
{{Paper
|id=None
|storemode=property
|title=Summary of the BPI Challenge 2013
|pdfUrl=https://ceur-ws.org/Vol-1052/summary.pdf
|volume=Vol-1052
|dblpUrl=https://dblp.org/rec/conf/bpm/X13
}}
==Summary of the BPI Challenge 2013==
https://ceur-ws.org/Vol-1052/summary.pdf
Business Process Intelligence Challenge 2013
B.F. van Dongen1 , B. Weber2 , D.R. Ferreira3 , and J. De Weerdt4
1
Eindhoven University of Technology,
Eindhoven, The Netherlands
b.f.v.dongen@tue.nl
2
Universität Innsbruck,
Innsbruck, Austria
barbara.weber@uibk.ac.at
3
Technical University of Lisbon,
Lisbon, Portugal
diogo.ferreira@ist.utl.pt
4
Queensland University of Technology,
Brisbane, Australia
jochen.deweerdt@qut.edu.au
1 Introduction
For the third time, the Business Process Intelligence workshop hosted the Busi-
ness Process Intelligence Challenge. The goal of this challenge is twofold. On
the one hand, the challenge allows researchers and practitioners in the field to
show their analytical capabilities to a broader audience. On the other hand, the
challenge (and it’s data) allows for researchers to prove that their techniques
work on real-life data sets.
Every year, the challenge organizer’s look for a real-life event log which con-
tains event-data of one or more operational business processes of an organization.
This data is provided to the participants as-is, without any pre-processing or fil-
tering (other than anonymization). The logs are made publically available and
are given a DOI for future reference.
In contrast to authors of scientific papers, challenge participants are not asked
to write scientific descriptions of algorithms, techniques or tools, nor are they
asked to provide scientifically well set-up case studies. Instead, the participants
are asked to analyze the provided log data using whatever techniques available,
focusing on one or more of the process owner’s questions or proving other unique
insights into the process captured in the event log.
A jury consisting of academic and industry members with a strong back-
ground in business process analysis assesses the submitted reports on complete-
ness of the analysis, presentation of the results and on originality. Finally, the
jury decides on a winner who receives a prize offered by one of the challenge’s
sponsors:
Fluxicon - Process mining for professionals. The process mining technol-
ogy in Fluxicon’s products can automatically create smart flow diagrams of
your process. All you need are event logs that are already on your IT sys-
2 B.F. van Dongen et al.
tems. Because our products work with this objective information, you no
longer need to rely on belief or hearsay you will know what’s going on.
Perceptive Software. Through the recent purchase of Pallas Athena, Percep-
tive Software has become a world-leading Business Process Management Soft-
ware (BPM) and Solutions provider. Their innovative software platforms and
user-friendly designs are well known and recognized throughout the industry.
2 The Event Log of Volvo IT Belgium
For the 2013 edition of the BPI challenge an event log was provided by Volvo IT
Belgium. The log contains events from an incident and a problem management
system called VINST. The primary goal of the incident management process
is restoring a customer’s normal service operation as quickly as possible when
incidents arise ensuring that the best possible levels of service quality and avail-
ability are maintained. The problem management system includes the activities
required to diagnose the root cause(s) of incidents and to secure the resolution
of those problems to enhance the quality of ITservices delivered and/or operated
by Volvo IT.
The data contained events related to the two processes as well as different
organisational entities and was separated into three event logs:
Incidents [1]. A log of 7554 cases containing 65533 events pertaining to the
incident management process.
Closed Problems [2]. A log of 1487 cases containing 6660 events pertaining
to closed problems in the problem management process.
Open Problems [3]. A log of 819 cases containing 2351 events pertaining to
open problems in the problem management process.
The process owner was particularly interested in four different questions.
Push to Front (incidents only). Is there evidence that cases are pushed to
the 2nd and 3rd line too often or too soon?
Ping Pong Behavior. How often do cases ping pong between teams and which
teams are more or less involved in ping-ponging?
Wait User abuse. Is the “wait user” substatus abused to hide problems with
the total resolution time?
Process Conformity per Organisation. Where do the two IT organisations
differ and why?
3 Submissions
While the first BPI challenge in 2011 attracted only three submissions and the
second BPI challenge in 2012 attracted six, a total of 12 submissions were re-
ceived for assessment by the jury this year. In this section, all submissions are
presented. For each submission, a short abstract is included followed by a selec-
tion of the jury’s comments on that submission in italics.
3rd Business Process Intelligence Challenge, 2013 3
M. Arias and E. Rojas, Pontificia Universidad Católica de Chile,
Chile [4]
This essay focusses on a couple of the process owners questions. The authors
of [4] present an analysis realized through applying different kinds of tools and
process mining techniques. They provide an analysis, which discoveres behavior
characteristics associated with products, resources and organizational lines.
The Jury: This is a solid analysis covering many details of the analysis. The
report illustrates the method of analysis coming from general to more refined
questions that are answered step by step, typically through filtering and process
discovery or other kinds of analysis. The report also highlights problems with
existing tools and how they have been resolved.
A. Bautista, S. Akbar, A. Alvarez, T. Metzger and M. Reaves, CKM
Advisors, USA [5]
The goal of this study is to identify opportunities that improve operational per-
formance of information technology incident management at Volvo, Belgium.
Findings are derived exclusively from computational analysis of the event logs.
Improvements that increase resource efficiency and reduce incident resolution
times and subsequently customer impacts are identified across the following ar-
eas: service level push-to-front, ping pong between support teams, and Wait-User
status abuse. Specific products, support teams, organizational structures, and
process elements most appropriate for further study are identified and specific
analyses are recommended.
The Jury: It is very interesting to analyze the data at product and country
granularities, in order to suggest possible candidates as root causes of bottlenecks.
In general, the number of variables taken into account is remarkable and this
suggests long and accurate work on actual data.
S. vanden Broucke, J. Vanthienen and B. Baesens, KU Leuven,
Belgium and University of Southampton, UK [6]
This report presents results related to the following investigations. First, an
open-minded exploratory analysis of the given event logs and second, answering
the four specific questions posed by the process owner. To do so, the authors
utilize both already existing as well as dedicated developed tools, and heavily
combine traditional data analysis tools and process-oriented techniques. They
indicate the existence of a gap between these two categories of tools and as such
emphasize the importance of a hybrid approach in a process intelligence context
throughout the report.
4 B.F. van Dongen et al.
The Jury: I particularly liked how carefully and consistently the potential
biases were revealed, and that interpretations (particularly concerning individual
employees) were given with caution. It’s very important to keep in mind that
the actual root cause of what can be observed behind a process often lies outside
of the data analyst’s view (e.g., “Is this team slow or just deals with the more
complicated cases?”).
E. Dudok and P. van den Brand, Perceptive Software, The
Netherlands [7]
This paper describes the results of the exploratory process mining efforts on the
incident management process. Specific areas of interest provided by the process
owner are analyzed as well as some additional areas of interest that qualified
for further investigation based on the information provided. Interesting results
include uncovering specific support teams and products for which specified un-
wanted behavior such as lack of push to front, ping pong, and wait user abuse
was prominent. Also some interesting relations were found, e.g. between the wait
user abuse and incident impact category, and the hypothesis that a correlation
exists between the number of handovers and total resolution time was proven.
The Jury: The submission addresses all questions of the process owner in a
convincing and very detailed manner. This report goes deep into the different
questions and presents findings in such a way that that they are accessible to
readers without deep process mining knowledge. In addition, conclusions are ex-
plained not at a technical level, but a business level. Most important conclusions
are briefly summarized at the end of each question. In addition, an executive sum-
mary is provided at the end of the document summarizing the main insights and
outlining potential additional directions of analysis. In addition to the questions
of the Process Owner an additional area of analysis (Resolution Verification) is
added.
F. van Geffen and R. Niks, Rabobank, The Netherlands and O&I
Management Consultants, The Netherlands [8]
Process mining is an accessible technique to visualize and analyze process vari-
ation and to yield improvements. The authors experienced that Process Mining
can help to overcome some of the barriers of the Six Sigma DMAIC cycle in
improvement projects. This results in a significant acceleration to complete such
a cycle.
The Jury: The report is less of a classical solution to the BPI challenge,
but a case study on conducting a DMAIC (Define, Measure, Analyze, Improve,
Control) Analysis from the Six Sigma Toolkit in a faster and more objective way
through the use of Process Mining tools. The BPI challenge data is used as an
exemplary case study to support this idea. The paper is a nice testimonial about
the usefulness of process mining techniques in the context of BPM improvements
efforts such as Six Sigma.
3rd Business Process Intelligence Challenge, 2013 5
J. Hansen, ChangeGroup, Denmark [9]
The purpose of this document is to answer the questions raised by Volvo IT
Belgium. In addition, an attempt is made to capture the incident and problem
management processes in the form of BPMN models.
The Jury: The submission addresses all questions of the process owner us-
ing Disco, Excel, Word and Enterprise Architect. The report does not comprise
any additional analyses, but provides process documentation in form of BPMN
models.
J. Hevia and C. Saint-Pierre, Pontificia Universidad Católica de
Chile, Chile [10]
In this work, the authors tried to give response to the client concerns and provide
analysis based on the data, with proposals to improve the performance of the
processes in the company. The paper attempts to identify the impact of various
failures and the organizaton of the process so that in the future Volvo can correct
and thus, provide a better service to it’s customers.
The Jury: In general, this report tends to make excessive use of process min-
ing techniques even when simple data analysis would suffice to answer the clients
questions. However, in one particular instance, such approach allowed the au-
thors to discover unexpected behaviour which may be important for the client to
know (regardless of their initial questions). This is the case of the push-to-front
question, where the authors discovered an escalation of some incidents from level
1 directly to level 3, which might not be desirable.
C. J. Kang, Y. S. Kang, Y. S. Lee, S. Noh, H. C. Kim, W. C. Lim, J.
Kim and R. Hong, Myongji University, Korea [11]
Recently, there has been a strong interest in the application of innovative pro-
cess mining techniques to promote evidence-based understanding and analysis of
organizations business processes. Following the trend, this report analyzes the
challenge logs. To create relevant datasets for answering the given questions,
the logs are pre-processed with the help of PL-SQL and Java. The datasets
are analyzed using ProM’s and Disco’s state-of-the-art process mining capabil-
ities, SQL, and traditional spreadsheet-based techniques. The authors provide
evidence-based answers to the questions and demonstrate the potential benefits
of process mining-based understanding and analysis of business processes.
The Jury: I liked the analysis and found the reasoning style easy to follow.
The authors made a point of clearly defining their interpre-tations and arguing
for their assumptions. The graphs were mostly well-explained. The solution to
process conformity per organization is perfect as far as I know.
6 B.F. van Dongen et al.
J. Martens, Capgemini, The Netherlands [12]
A professional application of Process Mining has been established in the context
of a methodology as defined by a consultancy firm. The results of the research
show where in the context of consultancy Process Mining is used and how clients
can benefit from expertise and standardized work.
The Jury: This report attempts to diligently answer, one by one, all questions
from the client. I appreciate a consultant taking time to perform such a report,
and I do encourage the author to continue in this direction.
Z. Paszkiewicz and W. Picard, Poznan Unviersity of Economics,
Poland [13]
In this paper, the authors provide answers to all the process owner’s questions
using process mining and social network analysis techniques, and they state the
existence of hidden support lines degrading the overall performance of incident
handling, little localized ping-pong behavior and wait-user misuse, and various
levels of conformity across organizations.
The Jury: The authors frequently state an hypothesis and then prove or dis-
prove it. This is very good. There is an interesting mix of tools deployed, data
inconsistencies are addressed, and assumptions are clearly stated. It’s a detailed
and good-quality analysis. The chord diagrams are an innovative and interesting
contribution.
S. Radhakrishnan and G. Anantha, SolutioNXT Inc., US [14]
The goal of this paper is to identify some key actionable patterns for improve-
ment. The authors have used a combination of process discovery tools (such as
Disco) and reusable scripting on MS Excel to perform their analysis. The focus
of their approach is to discern findings and encapsulate them within real world
perspectives. The authors brought this real world perspective by reclassifying
the given dataset into a) All cases b) Incidents only b) Incidents escalated to
problems and c) Problems only. They assessed a) wait status abuse, b) ping
-pong behavior across levels and across teams and c) general case flow pattern.
They uncovered interesting finding and captured a set of clear recommendations
based on these findings.
The Jury: All the analyses are performed with Disco and a graphical fuzzy
model is used to respond to all questions. The work finishes with three interesting
business-level suggestions.
3rd Business Process Intelligence Challenge, 2013 7
P. Van den Spiegel, L. Dieltjens and L. Blevi, KPMG Advisory,
Belgium [15]
The incident and problem management process forms an essential part in every
organization. Since businesses rely heavily on IT, each outage, issue or user
service request should be dealt with as quickly as possible in order to minimize
its impact on operations. The authors of this report objectively verified the
efficiency and effectiveness of the underlying process. The analysis was performed
by means of a of a combination of process mining and data mining techniques
and tools, including Disco, ProM, Minitab and MS Excel. As part of the exercise,
aspects such as total resolution times of tickets, actual resolution process being
followed, ping-pong behavior between the different helpdesk lines, differences
between distinct support teams etc. are investicated. Finally, recommendations
to improve the current process and increase integration between incident and
problem management are provided.
The Jury: I like the mapping of statuses on the standard flow. This really
bridges the gap between the data in the event log and the provided process model.
Furthermore, the authors make good use of their experience in the business do-
main and include interpretations from a business level perspective beyond the
questions asked by the process owner in the case description. Overall, it’s a solid
contribution with a good focus on business level insights.
4 The Winner
Using the jury’s scores as initial ranking of the 12 submissions, we obtained a
clear top three:
– A. Bautista, S. Akbar, A. Alvarez, T. Metzger and M. Reaves, CKM Advisors,
USA
– E. Dudok and P. van den Brand, Perceptive Software, The Netherlands
– C.J. Kang, Y.S. Kang, Y.S. Lee, S. Noh, H.C. Kim, W.C. Lim, J. Kim and R.
Hong, Myongji University, Korea
All these submissions were praised for their thoroughness, completeness and
presentation. However, one winner had to be selected and the decision was made
to select, as winner of the BPI Challenge 2013:
C.J. Kang, Y.S. Kang, Y.S. Lee, S. Noh,
H.C. Kim, W.C. Lim, J. Kim and R. Hong,
Myongji University, Korea
Their report was found to be complete, repeatable and thorough, while main-
taining a proper mix between general data analysis techniques and real business
process intelligence techniques. During the BPI workshop 2013, the prizes were
handed over to the first author of the submission as shown in Figure 1.
8 B.F. van Dongen et al.
Fig. 1. Award ceremony at BPI’13 showing Prof. Kang (right) receiving the BPI 2013
trophy from the organizers.
5 Conclusion
The BPI challenge has proven to be a succesful way to let practitioners and
researchers come together and share their capabilities and techniques. The vari-
ous reports have significantly increased in quality over the years and this year’s
result shows how mature the business process intelligence field has become.
With the help of the community, we hope to organize many succesful chal-
lenges in the future.
References
1. BPI Challenge 2013, incidents. Ghent University. Dataset.
http://dx.doi.org/10.4121/uuid:500573e6-accc-4b0c-9576-aa5468b10cee
2. BPI Challenge 2013, closed problems. Ghent University. Dataset.
http://dx.doi.org/10.4121/uuid:c2c3b154-ab26-4b31-a0e8-8f2350ddac11
3. BPI Challenge 2013, open problems. Ghent University. Dataset.
http://dx.doi.org/10.4121/uuid:3537c19d-6c64-4b1d-815d-915ab0e479da
4. M. Arias and E. Rojas, Pontificia Universidad Católica de Chile, Chile Volvo
Incident and Problem Management Behavior Analysis Proc. of 3rd BPI Chal-
lenge BPIC’13, Beijing, China, August 26, 2013, CEUR-WS.org, online http:
//www.CEUR-WS.org/Vol-1052/paper1.pdf
5. A. Bautista, S. Akbar, A. Alvarez, T. Metzger and M. Reaves, CKM Advisors, USA.
Process Mining in Information Technology Incident Management: A Case Study at
3rd Business Process Intelligence Challenge, 2013 9
Volvo Belgium Proc. of 3rd BPI Challenge BPIC’13, Beijing, China, August 26,
2013, CEUR-WS.org, online http://www.CEUR-WS.org/Vol-1052/paper2.pdf
6. S. vanden Broucke, J. Vanthienen and B. Baesens, KU Leuven, Belgium and Uni-
versity of Southampton, UK. Volvo IT Belgium VINST Proc. of 3rd BPI Chal-
lenge BPIC’13, Beijing, China, August 26, 2013, CEUR-WS.org, online http:
//www.CEUR-WS.org/Vol-1052/paper3.pdf
7. E. Dudok and P. van den Brand, Perceptive Software, The Netherlands. Min-
ing an Incident Management Process Proc. of 3rd BPI Challenge BPIC’13, Bei-
jing, China, August 26, 2013, CEUR-WS.org, online http://www.CEUR-WS.org/
Vol-1052/paper4.pdf
8. F. van Geffen and R. Niks, Rabobank, The Netherlands and O&I Management
Consultants, The Netherlands. Accelerate DMAIC using Process Mining Proc.
of 3rd BPI Challenge BPIC’13, Beijing, China, August 26, 2013, CEUR-WS.org,
online http://www.CEUR-WS.org/Vol-1052/paper5.pdf
9. J. Hansen, ChangeGroup, Denmark. Analyzing Volvo IT Belgium’s Incident and
Problem Management Data Using Automated Business Process Discovery Proc.
of 3rd BPI Challenge BPIC’13, Beijing, China, August 26, 2013, CEUR-WS.org,
online http://www.CEUR-WS.org/Vol-1052/paper6.pdf
10. J. Hevia and C. Saint-Pierre, Pontificia Universidad Católica de Chile, Chile. Ana-
lyzing Volvo Information with Process Mining Proc. of 3rd BPI Challenge BPIC’13,
Beijing, China, August 26, 2013, CEUR-WS.org, online http://www.CEUR-WS.org/
Vol-1052/paper7.pdf
11. C. J. Kang, Y.S. Kang, Y.S. Lee, S. Noh, H.C. Kim, W.C. Lim, J. Kim and
R. Hong, Myongji University, Korea. Process Mining-based Understanding and
Analysis of Volvo IT’s Incident and Problem Management Processes Proc. of 3rd
BPI Challenge BPIC’13, Beijing, China, August 26, 2013, CEUR-WS.org, online
http://www.CEUR-WS.org/Vol-1052/paper8.pdf
12. J. Martens, Capgemini, The Netherlands. Professional Use of Process Mining
for Analyzing Business Processes Proc. of 3rd BPI Challenge BPIC’13, Bei-
jing, China, August 26, 2013, CEUR-WS.org, online http://www.CEUR-WS.org/
Vol-1052/paper9.pdf
13. Z. Paszkiewicz and W. Picard, Poznan University of Economics, Poland. Anal-
ysis of the Volvo IT Incident and Problem Handling Processes using Process
Mining and Social Network Analysis Proc. of 3rd BPI Challenge BPIC’13, Bei-
jing, China, August 26, 2013, CEUR-WS.org, online http://www.CEUR-WS.org/
Vol-1052/paper10.pdf
14. S. Radhakrishnan and G. Anantha, SolutioNXT Inc., US. Process Improvement
Focused Analysis of VINST IT Support Logs Proc. of 3rd BPI Challenge BPIC’13,
Beijing, China, August 26, 2013, CEUR-WS.org, online http://www.CEUR-WS.org/
Vol-1052/paper11.pdf
15. P. Van den Spiegel, L. Dieltjens and L. Blevi, KPMG Advisory, Belgium. Applied
Process Mining Techniques for Incident and Problem Management Proc. of 3rd
BPI Challenge BPIC’13, Beijing, China, August 26, 2013, CEUR-WS.org, online
http://www.CEUR-WS.org/Vol-1052/paper3.pdf