Process mining is a field of research that has gained much attention in recent years because of its ability to analyze and improve processes. Indeed, one of the key aspects of process mining is its ability to predict the activities in the future and the time spent on these activities. In this work is proposed the use of Bidirectional LSTM and Multi-Speed Transformer on a recent dataset called BPIC-2020 related to reimbursement process of the University of Technology of Eidenhoven. Results shows that Multi-Speed Transformer is more capable to performs next activity prediction than the Bi-LSTM. Meanwhile, for the elapsed time prediction, the viceversa is true.
Process Mining is a research domain that gains
a lot of interest thanks to the application of Data
Mining and Machine Learning methods to the
processes [1]–[3]. The application is performed
on processes recorded as timeseries data called
“event log”, accordingly to [4]. In this way several
analysis can be performed, starting from an initial
discovery study [4] (building new models of the
processes recorded) to a conformance and
modeling check to analyze the evolving situation
of processes and, thus, align models, and related
software, to business processes [
The analysis of timeseries is a problem broadly manage using several types of models. Starting from the using of classical LSTM [2], [3] to the use of the Convolutional Neural Networks [1], [6].
The importance of the Process Mining techniques is also highlighted by the Public Administration (PA) that starts to use such technology in order to build models that helps to enhance the quality of the PA’s processes [7]. The application of Process Mining starts from the process modelling, using several techniques as design processes based on collected data [8], or analysing processes in order to improve them [9].
The PA involved in the use of Process Mining techniques is not only related to governance[7]– [9], but also related to health [10], [11] and educational system [12], [13].
In this work, it is proposed the use of an neural network architecture depicted in Figure 1. The architecture is based on the use of “block” meant to be the same neural network architecture used firstly to analysis the timeseries and then using a block to predict the next activity and the other one to predict the elapsed time to the next activity. This model was presented in [2] using a BiLSTM. The work is structure presented related works in Section 2. Then the dataset and the used method are explained in section 3. The experimental set-ups to perform experiments are explained in Section 4. Results and their discussion are presented in Section 5. Finally in of the University of Eindhoven (TU/e). The data
The analysis of the event logs can be performed for several reason. In this work, it was proposed the use of the architecture presented by Gunnarsson B et al in [2]. The main architecture is depicted in Figure 1. It is noticeable that such architecture is based on the use of three blocks that represents the same underlying neural networks. In this way, a first application performs a sort of features extraction from the timeseries, meanwhile the other two application are specialized to predict the next activity and the elapsed time to the next activity. As in the majority of works using timeseries (called also event log), in [2] it was proposed the use of a
(Bi-LSTM) in order to capture temporal patterns looking forward and backward.
Dentamaro V. et al, propose a new architecture called
architecture is based on a firstly application of a multi branch analysis in order to analysis with different level of details the data. In this way, analogously to the use of microscopy, it possible to identify fine patterns and more gross patterns.
The
Transformer using information improved the State of the Art in terms of prediction metrics.
Hence, in this work is proposed the use of both 3.
In this section the used methods to perform experiments about next activity and elapsed time prediction are explained along with the used dataset. 3.1
In this work, it is used a dataset related to the activity performed by University of Technology of Eindhoven (TU/e). In particular, the dataset that comes from the Business Process Intelligence Challenge 2020 (BPIC-2020) [14]. were collected from the 2017 to the 2018.
Furthermore, the collected data are organized in several event log. Specifically, the event logs are: • “Domestic Declarations” that is related to domestic travel (within the same country). The event log related to “Domestic Declaration” contains 56437 events recorded and the recorded events are about 10500 cases; • “Request for Payement” that contains cases that could be not related to travels. The event log related to “Request for
Payment” contains 36796 events for 6886 cases; • “International Declarations” is related to travel outside the country. The event log contains 72151 events and the recorded events are about 6449 cases; • “Travel Permit Data” is related to the permission to travel. This event log is composed by 86581 events for 7065 cases; • “Prepaid Travel Costs” contains data related to the travel costs prepaided. This events log is composed 18246 events and the recorded events are related to 2099 cases.
Hence, each event log is composed with several events.
Within each event log is possible to identify and separate cases in order to obtain, for each case, a separate event log, called “trace”. The “trace” is used to preprocess data in order to apply methods explained in the next section. A “trace” is composed by a sequence of events with 1 < <
where is the number of events for the “trace” . The event is composed as a record of information as: “case id” , “activity label” , “timestamp” , “attributes j-th” where 1 < < and M is the number of attributes. The attributes can contains information related to the case (this information are shared from all the activity and they don’t change long the trace) and to the activity (this information are specific to the recorded activity). 3.2
In this subsection, the used methods to perform prediction about the next activity and the elapsed time are explained.
The used methods are based on the use of a new State of Art neural network called “MultiSpeed Transformer” [6] and a well-known Bidirection LSTM (referred also Bi-LSTM). This two neural networks were used as block in a more complex architecture in order to perform both next activity and elapsed time prediction.
The main architecture is presented in Figure 1.
The architecture takes as input a preprocessed trace as in Table 1. Hence, the sequence of prefixes is feed to the first block. In this way a first analysis is performed. The output of the first block is then passed to both “next activity block” and “elapsed time block”. Such blocks are equals to the first block, because they use the same structure. But the “next activity block” gives as ouput a distribution of probability to predict the next activity, meanwhile the “elapsed time block” gives the predicted elapsed time to the next activity. 4.1
Bidirectional LSTM is a neural network architecture based on the processing of timeseries. In particular, the core part is the “LSTM” that analyse data in order to find temporal patterns.
Commonly, such analysis is performed in forward setup, i.e. from the past to the present.
In Bidirectional LSTM, two layer of LSTM are used in order to analyse data in forward and backward way. In this way, both forward and backward temporal patterns are learnt by the model. 4.2
The Multi-Speed Transformer was presented by Dentamaro V. et al [6]. This model is based on the concept of fine and gross analysis of data. Such analysis are done at different levels of detail as analysis performed by microscopies at different resolutions. In this way, different kind of information are extracted and then concatenated to obtain the prediction.
In this section the setups used to perform experiments are explained. 5.1
Each event log is also divided in train and test split following the set-up used in [2]. Specifically, the cases are ordered by the timestamp of the related recorded event. Then a 75/25 split is applied. After such division, from the train set are eliminated the cases which are not ended when the In order to use the dataset BPIC2020, each trace, in each event log, is firstly sorted accordingly with the timestamps of each event, then it is preprocessed extracting information as “prefix”, “suffix” and “elapsed time”. In the following, this information are defined: • “Prefix”: Prefix is defined as a function that given a trace , a position and a windows size , it return a sequence of events from − to :
( , , ) = < { − }, … , { } > • “Suffix”: Suffix is also defined as a function that given a trace , a position and a windows size , it return a sequence of events from + 1 to + :
( , , ) = < { +1}, … , { + } > • “Elapsed Time”: Elapsed time is the time remaining to the next activity. Hence it is defined as a function that given a trace , a position it returns the difference between the timestamp of +1 and :
( , ) = +1. − . The “.” is intended as the operator that recall the timestamp values of the event. The elapsed time is computed in seconds.
For both “prefix” and “suffix”, if the windows size excides the number of events to be selected zero-padding is applied, e.g. for the prefix, if the given position is lower than the windows size it means that “k-w” is negative, hence to overcome this problem zeros are added.
Finally, in the Table 1 it is shown an example of the results obtained from the preprocessing. It is supposed that the given trace is composed by 5 events: = < 1, 2, 3, 4, 5 >, and the windows size is = 3:
Table 1 Example of preprocessing of a trace. For the Suffix the windows size is 1 to obtain only the next activity to predict
Prefix(T,p,w)
Suffix(T,p,1)
Elapse Time(T,p) 2 − 1 3 − 2 4 − 3 5 − 4 t w first case of the test set started. Successively, the train set is further divided in train and validation set in 75/25 setup.
For each event log, the activity label are one hot encoded.
Once the experiments were performed, predictions were evaluated accordingly to the following metrics: • Categorical Accuracy: The proportion of the correctly predicted “next activity label” on the total number of predicted “next activity label”. • Root Mean Squared Error: The root applied at the sum of the squared differences between predicted “elapsed time” and real “elapsed time”.
The results of the performed experiments are reported in Table 2: From the results showed in Table 2 it is possible to notice the in terms of accuracy to predict the next activity, the proposed use of the Multi-Speed Transformer is better than the proposed use of the Bidirectional LSTM (Bi-LSTM). Indeed, the best performance of the Bi-LSTM is 10% lower than the lower performance using the Multi-Speed Transformer.
Related to the elapsed time, it is possible to notice that the Root Mean Square Error (RMSE) is lower for the models using the Bi-LSTM. In this sense, the model using the Bi-LSTM seems to be more suitable to predict the elapsed time to the next activity.
A more deeply analysis of the results highlights that the model using Bi-LSTM is truly better than the model using Multi-Speed Transformer to predict elapsed time to the next-activity, but the RMSE in all the models and for all the used event log are high.
In this work, the use of Bi-LSTM is compared with the use of Multi-Speed Transformer as block of a more complex architecture to build systems capable of both prediction of the next activity and elapsed time to the next activity.
The results shows that multi-speed Transformer is suitable to predict next activity given a sequence of activity. Meanwhile, the prediction of the elapsed time seems to be a task more suitable for the model using the Bi-LSTM.
Future works could include more information about the case or the activity in order to improve the quality of predictions.
This Word template was created by Aleksandr Ometov, TAU, Finland. The template is made available under a Creative Commons License Attribution-ShareAlike 4.0 International (CC BYSA 4.0).
[1] [2] [3]
E. Obodoekwe, X. Fang, and K. Lu, “Convolutional Neural Networks in Process Mining and Data Analytics for Prediction Accuracy,” Electronics (Switzerland), vol. 11, no. 14, Jul. 2022, doi: 10.3390/ELECTRONICS11142128. B. R. Gunnarsson, S. vanden Broucke, and J. De Weerdt, “A Direct Data Aware LSTM Neural Network Architecture for Complete Remaining Trace and Runtime Prediction,” IEEE Trans Serv Comput, 2023, doi: 10.1109/TSC.2023.3245726. N. Tax, I. Verenich, M. La Rosa, and M. Dumas, “Predictive Business Process Monitoring with LSTM Neural Networks,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture