An often investigated subject at the intersection of Business Process Management (BPM) and the Internet of Things (IoT) is the abstraction of low-level IoT events to BPM-level activities [1], which can be seen as a multi-class activity detection problem. In previous work, we presented a corresponding method [2] that has the goal of detecting business process activities in real-time, based on annotated IoT data streams to enable online process conformance checking [3]. While investigating methods to evaluate the quality of the IoT-based detection of activities, which we capture in event logs in XES [4] format, we realized that most event log comparison tools that exist in the BPM field are not suitable to evaluate activity detection methods. They do not provide helpful metrics for the comparison of a detected event log (e.g., created from IoT data by the detection method) with a ground truth event log (representing the correct sequence and timing of activities as manually annotated or predefined) to evaluate and improve a specific detection method, since they i) focus on variant comparison to derive insights for process analysts regarding business outcomes (i.e., comparing process performance indicators) [5]; and/or ii) they produce results that are not suitable for rapid comparison due to non-quantitative outputs (e.g., graphs or natural language) [5]. Thus, we sought out methods from other relevant fields in the literature.
Core requirements for the event log comparison tools and metrics derived from our usecases [3] are: i) they need to provide insights relevant for the detection quality of an activity detection method, i.e., on whether the activities are detected to be in-/active at the right times and/or whether the sequence of detected activities is correct w.r.t. a given ground truth; ii) they need to provide quantitative results for rapid and automatable comparison (e.g., for programmatic exploration of a potentially large number of method parameters); and iii) they should provide insights over multiple cases within the event logs. We found multiple suitable metrics in the areas of information theory, signal processing, and general activity recognition. However, none of them could be directly applied to BPM-related concepts (i.e., XES-based event logs): some important metrics (e.g., from [6]) were not available as open implementations, and some others needed modification to make sense in the context of BPM (e.g., cross-correlation). Thus, we decided to implement, modify, extend, and integrate them in a tool ourselves.
The result is the Python library AquDeM1 : Activity and Sequence Detection Evaluation Metrics, which takes two event logs in XES format as an input-one ground truth (GT) log and one log containing the detected (DET) activities-and allows for the calculation of a variety of comparison metrics for evaluation. Besides the library as core contribution, we have created a web application that utilizes the library, allowing for quick, intuitive, visual comparison of two event logs. In our research [2,3], the Python library is not only used in this web app, but also in other more automated pipelines. The separation into library and web app allows for more varied use-cases without impacting the functionality or usability of either.
The metrics available in AquDeM can be categorized into activity level metrics and sequence level metrics. Activity level metrics are calculated for each activity type in each case separately. A sequence level metric is calculated for each case separately, but over all activity types in that case. For calculations that span multiple cases/activities, the results are aggregated, currently using the mean. Another categorization is into frame-based or event-based metrics [cf. 6]. Frame-based metrics are calculated based on specific time points when an activity is detected as (in-)active, making them dependent on the (IoT) data's sampling frequency (i.e., how often data is recorded). Event-based metrics work on the classification of events themselves and do not take the sampling frequency into account. For the calculation of the metrics, the event logs must minimally adhere to these requirements: i) each activity execution needs both a start and a complete event; and ii) the logs need a sampling frequency for the frame metrics.
The metrics were selected from literature, implemented and modified based on the requirements in Section 1. To get a better understanding of the metrics, we provide intuitive (noncomplete) explanations below. Note, that all of these metrics are also available in normalized form in the library, allowing for comparison among different event logs. In Table 1 we give an overview of the metrics regarding the categories described above, together with references to their definitions. Furthermore, we provide a usage example of AquDem in Listing 1.
• Cross Correlation (CC) measures the similarity between the DET and GT time series by determining the shift at which they are most alike and quantifying that similarity, relative to perfect equality for time series of that length.
Two Set (TS) metrics classify frames into categories such as true positive, true negative, deletions, fragmentations, mergings, insertions, and over-fillings or under-fillings at the start and end of an activity instance.
• Event Analysis (EA) metrics categorize the GT events as correct, deleted, fragmented, merged, or both fragmented and merged; and DET events as correct, inserted, fragmenting, merging, or both fragmenting and merging.
• The Levenshtein-Distance (LD) calculates the minimum number of single activity instance edits (insertions, deletions, or substitutions) needed to transform the sequence of activity instances in DET to match GT.
• The Damerau-Levenshtein-Distance (DLD) extends the LD metric by also considering the transposition of two adjacent activity instances as a single edit.
The web app, built using streamlit,2 has proven to be useful for the iterative and exploratory process of evaluation and development of the detection method in [2]. Notably, the library has been developed in tandem with the web app: it is built with interactive and repeated calculations in mind, i.e., browsing metrics and going back-and-forth with different analysis parameters. The library internally relies on caching to speed up recurring requests and to re-use computations from previous requests for similar requests (e.g., a filtered view that contains data calculated in a previous view). Screenshots of the web app can be seen in Figure 1. After uploading two XES logs, the user can choose a certain metric to visualize. The visualization, tabular presentation, and further options for filtering are varied for each metric to offer suitable presentations for exploration with that particular metric. The app provides specific visualizations we deemed useful (based on our use-cases), with a more flexible exploration being possible with the library.
small-synth: synthetic logs, 2 cases each and 34 start / complete events in total medium-exp: experimental logs, 1 case each and 262 start / complete events in total large-exp: experimental logs, 4 cases each and 1290 start / complete events in total Machine: CPU cores: 16; Model: AMD Ryzen 7 7840HS with Radeon 780M Graphics; Threads per core: 2; RAM: 32.0 GiB We consider the maturity of the library and web app to be relatively high: they are continually improved and extended as they are used in our research, and include an automated test suite with combined branch and line coverage of > 90%. To better understand the library from a performance and usability perspective, we have measured runtimes with a variety of event log pairs (GT and DET), including experimental logs from the smart factory scenario described in [2,3]. The results can be seen in Figure 2, indicating acceptable performance and scalability.
In this work we presented AquDeM: a tool featuring activity and sequence detection evaluation metrics to be used for event log comparison by BPM researchers. Besides the main, programmatically usable Python library, we provide a web app for fast, visual comparison of two event logs. The modular and decoupled design of library and web app allows for flexible usage. Given the increasing research attention in the area of activity detection in BPM and the absence of appropriate tools, we believe this to be a valuable addition to the pool of community resources.
This work has received funding from the Swiss National Science Foundation under Grant No. IZSTZ0_208497 (ProAmbitIon project).