Process mining is employed by organizations to completely understand and improve their processes and to detect possible deviations from expected behavior. Process discovery uses event logs as input data, which describe the times of the actions that occur the traces. Currently, Internet-of-Things environments generate massive distributed and not always structured data, which brings about new complex scenarios since data must first be transformed in order to be handled by process mining tools. This paper shows the success case of application of a solution that permits the transformation of complex semi-structured data of an assembly-aircraft process in order to create event logs that can be managed by the process mining paradigm. A Domain-Specific Language and a prototype have been implemented to facilitate the extraction of data into the unified traces of an event log. The implementation performed has been applied within a project in the aeronautic industry, and promising results have been obtained of the log extraction for the discovery of processes and the resulting improvement of the assembly-aircraft process.
Process mining facilitates the understanding of business processes of
organizations. This technique usually involves [
One of the challenges in process mining is related to the ability to extract
suitable events [
Aircraft
Raw Log
Raw Log WorkStation A
Event Log
Raw Log WorkStation B
Raw Log
WorkStation E
Raw Log WorkStation C
WorkStation D
Process Mining
Techniques structured schema. It brings about an extra level of complexity to be managed by the current process mining solutions that involve the use of event logs because they are mainly focused on structured homogeneous data.
The main issue tackled in this paper concerns how complex data structures, typically generated in IoT environments, can be transformed to create an event log that could be managed by the existing tools in process mining. Figure 1 shows a scenario of log extraction in the aeronautic industry, based on the data produced by workstations in an aircraft-assembly process. In this scenario, several workstations perform tests and produce logs of their execution. The information, which follows a complex data structure, is stored in a NoSQL database (i.e., MongoDB). The challenges are thus aggregating such complex data and extracting event logs in XES format from them. Then, process discovery techniques can be applied.
Hence, the objective of this paper is to develop an approach that enables the extraction of event logs in XES format from complex data. In addition, this solution is integrated in a tool so that non-expert users can benefit from this solution. 2
Nowadays, IoT environments are becoming highly relevant for organizations [
Event Log (XES) Extraction recipe
Event Log (XES)
This describes a complex process that is executed at several locations without
guidance from a Business Process Management System (BPMS) [
In this context, the main process in the organization is focused on the assembly of the aircraft in accordance with the design of an assembly chain formed of various workstations. Each workstation in the factory executes its own separate tests depending on the equipment that it has available, as described in the following: 1. An aircraft with an identifier (accode) is located at a workstation. 2. In each workstation, a set of test instructions must be carried out (gticode). 3. Each set of test instructions can be executed several times until it is validated by a designated operator (executor), and it begins and finishes at a specific moment (start_date, final_date) and successdate. 4. Each execution might either finish successfully or generate incidents. Those include an incidentcode, incidenttype, start_date, resolution_date, and label.
The workstations are physically distributed and generate complex data structures depending on the tests executed thereon. In order to generate logs that can be consumed by process-mining techniques, an extraction from the structure presented in Figure 2 must be defined. The left side of the figure represents a set of datasets formed of complex structures (lists and nested structures). However, the event logs must fit the schema presented in the right-hand side of the figure.
By using this data, several trace logs can be generated, which will depend on the type of process that must be discovered, for example, the evolution of the aircraft per workstation, the evolution of tests, or the life-cycle of incidents. These different perspectives (points of view) of the data can be used as the input of the process-discovery techniques to ascertain the actual process and to enable the analysis and improvement of the processes of assembly and testing.
The main problem to be faced in this scenario is the lack of proposals to extract
event logs from complex semi-structured data. The majority of the approaches
that focus on the extraction of event logs have been developed using relational
databases as a starting point. The main idea in these cases is that events leave
footprints by changing the underlying databases that are registered in the redo
logs of the database system [
Nonetheless, as the scenario of IoT confirms, relational databases are not
considered as the unique source of events. In [
To the best of our knowledge, our proposal constitutes the first approach for the construction of XES event logs from complex semi-structured data, abstracting non-expert users from the definition of complex transformations required to obtain a XES event logs. 3
First of all, the characteristics of the dataset (i.e., raw logs) for the real scenario
are depicted. In order to tackle the problems of event log extraction, we have
developed a Domain-Specific Language (DSL) [
In order to measure the complexity of the scenario, it is necessary to ascertain the characteristics of the datasets (i.e., raw logs) used as the input of our approach. As shown in Table 1, the dataset contains 9367 sets of tests, provided by 52 workstations. In these workstations, 15 aircraft have been tested, with a total number of 1; 110 tests (i.e., GTI). The tests can be executed more than once for each aircraft. This is the reason why there are 9397 different executions of these tests. These repetitions occur when a test execution is unsuccessful, thereby causing, an incident, hence there are 6; 049 incidents in the raw log. In order to complement this information, some statistical data has been extracted (for instance, the average of GTI per aircraft) that provide a greater level of detail. We would like to point out that the dataset presented is a subset of the real data obtained in the project Clean Sky 2. Our dataset is just a small part of the large amount of information generated, is made up of 9397 raw logs, filtered and modified.
Taking into account the IoT scenario, each workstation periodically provides a raw log on the execution of a test on a particular aircraft, which may also contain incidents. Example 1.1 illustrates a use of the DSL. It represents the process of the assembly aircraft for each of the workstations where the tests are evaluated. The aircraft production follows a process where each part thereof is assembled and tested in accordance with the design of the engineers. Each step of the assembly process is performed in a workstation, where the resulting event log is formed of a set of traces with the following information: 5 Due to confidentiality reasons, the data shown regarding incidents, time, dates have been manipulated and anonymised to avoid showing real data and the values only represent a subset of the real data. – case_id : accode (i.e., the aircraft identifier). – activity_id : workstation. Several tests can be executed in the same workstation, and therefore if more than one consecutive register of the testing of an aircraft is found in the same workstation, then only the first registration is considered in the creation of the event log. – timestamp : start_date (i.e., the date when the aircraft passed through the workstation).
The event log generated by the piece of code in Example 1.1 is shown in Example 1.2.
Example 1.1: Piece of extraction code extract( define trace id("accode"), define trace event( activity = "workstation", criteria = orderBy(t"start_date" -> toDate("MM/dd/yyyy HH:mm:ss")), timestamp = t"start_date" -> toDate("MM/dd/yyyy HH:mm:ss") This section describes how the obtained results help improve the process in the assembly-aircraft process. A tool implemented to validate the results is also present.
First of all, a set of test cases are outlined bellow. The results are discussed in next subsections.
– Test Case A. Process of the aircraft according to the workstation that executes the test. This test case has been described in Section 3 (cf. Example 1.1). – Test Case B. Process of the aircraft according to the GTI execution. During the aircraft assembly process, a set of test instructions must be carried out. In this test case, the processes of the aircraft according to the tests applied to them (gticode ) are extracted. Hence, the resulting event log has the form: case_id : accode (i.e., the aircraft identifier). activity_id : gticode. If an aircraft passes the same test more than once, then it is only considered the first time that it passed. timestamp : start_date (i.e., the first time the test was applied to the aircraft).
The piece of code which enables this test case to be carried out is shown in Example 1.3.
Example 1.3: Test Case B extract( define trace id("accode"), define trace event( activity = "gticode", criteria = orderBy(t"start_date" -> toDate("MM/dd/yyyy HH:mm:ss")), timestamp = t"start_date" -> toDate("MM/dd/yyyy HH:mm:ss") ) ) from ("mongodb://mongo-instance:27017/db/logs") – Test Case C. Process of the incidents according to the type of incident. As explained in Section 2, the tests (gticode ) could report a set of incidents with an (incidenttype ). The resulting event log must be in the following form: case_id : gticode (i.e., the test identifier). activity_id : incidents.incidenttype. If a type of incident is produced more than once in a test, then it is only considered the first time it was produced. timestamp : start_date (i.e., the first time this type of incident occurred).
The piece of code which corresponds to this extraction is shown in Example 1.4.
Once the event logs (i.e., XES files) are obtained, it is possible to perform certain studies on the data that they contain and discover the processes. Next, the logs retrieved after each extraction are explained in a more precise manner. Table 2 describes the features attained for each example of data extraction. A processdiscovery tool, Disco6, has been employed to analyse these logs.
The resulting processes discovered are illustrated in Figure 3. These processes help in the understanding and identification of certain information that could not be analysed manually. For instance, the process for Test Case A helps towards the understanding of the flow of the aircraft through the different workstations in the factory. We can observe how certain workstations can perform tests in parallel with other workstations. Test Case C can help in the understanding of the flow of the incidents, by showing how the incidents evolve.
The event logs and consequently discovered processes helped to improve the industry in the understanding and knowledge about the aircraft-assembly process:
Test Case A. In some cases an aircraft was in two workstations simultaneously, because for some tests it is easier to move the station than to move the aircraft. Regarding workstations, there are workstations with assumable transition values between them (hours or days) but also workstations in which the time in which the aircraft is at, is always 0 seconds, thus, we can assume that they are intermediate states in which nothing is actually done. On the other hand, there are workstations in which the aircraft has only been once and in almost all but one workstation the aircraft has been 0ms, in one of them (55
7 Due to confidentiality reasons, the data shown regarding incidents, time, dates have been manipulated and anonymised to avoid showing real data and the values only represent a subset of the real data. WorkStation
WorkStation
WorkStation
WorkStation
WorkStation
Test Case C Test Case B
Test Case A days). Knowing the average time invested in each workstation and the workstations through which the aircraft must pass for aircraft of the same type, it is possible for experts to know if deviations are occurring and the current state against what would be desired.
Test Case B. This case helped experts to know the sequence of tests that must be carried out and the time that must be spent in each one. Finally, it allows experts to make decisions on the improvements that could be applied to the tests performed on the workstations.
Test Case C. The discovery of the incident process by type makes it easier for experts to know which types of incidents are most likely to occur after another incident occurred. The discovery of the incident process by type helps know the average time that is likely to be invested in the resolution of the same incident and hence know how this will affect the entire assembly and testing process. The most frequent type of test is in all cases the first incident and also the last one. Other particular aspects are detected such as after the incident of shift change of operators, if an incident occurs, which occurs in more than 30 percent of cases, it is always of two possible types: Aircraft Configuration Failure or Test Edition Failure.
The implemented tool is based on three main components: (1) the DSL
language, which enables the specification of the recipes of extraction from NoSQL
databases, such as MongoDB, and the generation of event logs in XES format, (2)
a prototype of application (cf., Figure 4) in which transformation can be defined
in a user-friendly way, and; (3) a connector which allows the generated XES logs
to automatically feed the ProMTM8 tool for the discovery of process models. All
the resources that have been employed in this work are freely available at [
proposal has been also supported by the implementation of an instrument which is connected to process-mining tools.
The following important lessons have been learned in the development of the approach: i) extraction of event logs in IoT scenarios. The complexity of the data in IoT scenarios increases the complexity in the extraction of event logs that require a intensive analysis; ii) massive data production and processing. An environment where data is continuously being generated could require the integration with Big Data architectures; iii) the selection of case analysis useful for the organization. The selection of the case analysis and the activity filtering criteria can influence in the results of the selected case, therefore, it is not always an easy task since it involves a great pre-analysis of the data and a huge understanding of the organization, and ; iv) risk factors of discovery spaghetti-like processes. Depending on the data and the case of study the discovery process can retrieve a spaghetti-like process which increase the complexity of analysis of the discovery process.
This work has been partially funded by the Ministry of Science and Technology of Spain through ECLIPSE (RTI2018-094283-B-C33), the Junta de Andalucia via the PIRAMIDE and METAMORFOSIS projects, and the European Regional Development Fund (ERDF/FEDER). The authors would like to thank the Cátedra de Telefónica “Inteligencia en la Red“ of the Universidad de Sevilla for its support.