Automated Segmentation of User Interface Logs Using Trace Alignment Techniques (Extended Abstract) Simone Agostinelli Sapienza Università di Roma, Rome, Italy agostinelli@diag.uniroma1.it Abstract—Robotic Process Automation (RPA) is a fast- user actions associated with the routine itself in well bounded emerging automation technology that allows organizations to routine traces. A routine trace represents an execution instance automate high volume routines. RPA tools are able to capture of a routine within a UI log. To be more precise, starting in dedicated User Interface (UI) logs the execution of routines previously performed by a human user on the UI of a computer from a UI log previously recorded by a RPA tool and an system, and then emulate their enactment in place of the user interaction model representing the expected behaviour of a by means of a software (SW) robot. The issue to automatically routine performed during an interaction session with the UI, understand which user actions contribute to a specific routine we propose to leverage trace alignment in Process Mining [4] inside the UI log is also known as segmentation. The proposed to automatically identify and extract the routine traces by the research investigates how to leverage trace alignment techniques in Process Mining to automatically derive the boundaries of a UI log. Such traces are finally stored in a dedicated routine- routine by analyzing the UI log that keeps track of its execution, based log, which captures exactly all the user actions happened thus tackling the segmentation issue. during many different executions of the routine, thus achieving the segmentation task. I. I NTRODUCTION Robotic Process Automation (RPA) uses software robots II. S EGMENTATION U SING T RACE A LIGNMENT (or simply SW robots) to mimic and replicate the execution In this section, after providing the relevant background on of highly routine tasks (in the following, called routines) trace alignment (see Section II-A), we present a first approach performed by humans in their application’s User Interface to tackle the segmentation issue (see Section II-B). (UI). SW robots encode, by means of executable scripts, sequences of fine-grained interactions with a computer system. A. Alignment between UI Logs and Interaction Models Commercial RPA tools allow SW robots to automate a wide Trace alignment [4] is a conformance checking technique range of routines in a record-and-replay fashion. The current within Process Mining that is employed to replay the content practice for identifying the single steps of a routine is by of any trace of an event log against a process model repre- means of interviews, walk-throughs, and detailed observation sented as a Petri net, one event at a time. For each trace in of workers conducting their daily work. A recent approach the log, the technique identifies the closest corresponding trace proposed by Bosco et al. [1] makes this identification less that can be parsed by the model, i.e., an alignment, together time-consuming and error-prone, as it enables to automatically with a fitness value, which quantifies how much the trace extract from a UI log, which records the UI interactions during adheres to the process model. The fitness value can vary from a routine enactment, those routine steps to be automated with 0 to 1. A fitness value equals to 1 means a perfect matching a SW robot. While this approach is effective in case of UI between the trace and the model. logs that keep track of single routine executions, i.e., there In our context, we perform trace alignment by constructing is an exact 1:1 mapping among a recorded user action and an alignment γ of a UI log U (note that we can consider the specific routine it belongs to, it becomes inadequate when the entire content of the UI log as a single trace) and an the UI log records information about several routines whose interaction model w as a Petri net, which allows us to exactly actions are mixed in some order that reflects the particular pinpoint where deviations occur. To this aim, the events in U order of their execution by the user. In addition, since the same need to be related to transitions in the model. Building this user action may belong to different routines, the automated alignment requires to relate “moves” in the log to “moves” identification of those user actions that belong to a specific in the model. However, it may be that some of the moves in routine is far from being trivial. The challenge to automatically the log cannot be mimicked by the model and vice versa. A understand which user actions contribute to which routines move in log for a transition t indicates that t occurred when inside a UI log is also known as segmentation [2], [3]. not allowed; a move in model for a transition t indicates that In this research, we investigate a technique for automatically t did not occur, when, conversely, expected. Many alignments deriving the boundaries of a routine by analyzing the UI are possible for the same UI log and a Petri net. We aim log that keeps track of its execution, in order to cluster all at finding a complete alignment γ opt of U and w with Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). minimal number of deviations (i.e., of moves in log/model), alignment. At the end of the iteration, the routine-based log also known in literature as optimal alignments. For the sake of URw is stored into Uset , and the the next interaction model simplicity, we are assuming here that all the deviations have contained in Wset can be analyzed. In conclusion, a number of the same severity. However, the severity of a deviation can be routine-based logs equal to the number of interaction models customized on a ad-hoc basis [5]. under study are computed. B. A First Approach to Segmenting UI Logs III. D ISCUSSION , F UTURE W ORK AND C ONCLUSION The proposed approach underlying our segmentation tech- Our first solution to the segmentation issue is a supervised nique consists of two methodological phases, filtering and technique that leverages trace alignment to identify sequences trace alignment, to be applied in sequence. The envisioned of user actions in a UI log that belong to specific routine technique takes in input a UI log U , a set of interaction models executions, clustering them in well bounded routine traces. Wset and returns a set of routine-based logs Uset . For each Differently from event abstractions techniques [6], which map interaction model w ∈ Wset (one for each routine of interest) low-level event types to multiple high-level activities (while represented as Petri nets, the following steps are performed: the event instances, i.e., with a specific timestamp in the log, Filtering. The filtering phase is used to filter out noisy actions can be coupled with a single high-level activity), segmentation from the UI log. Specifically, for each interaction model techniques must enable to associate low-level event instances w ∈ Wset , a local copy of the UI log U w is created. Then, all (corresponding to our UI actions) to multiple routines. The user actions that appear in U w but that can not be replayed complete knowledge of the interaction models’ structure is, of by any transition t of w are removed from U w . The output course, the main limitation of the presented technique. of this step is a model-based filtered UI log Uφw . Working As a future work, we aim at relaxing the supervised with Uφw rather than with U w will allow us to apply the assumption in different ways: (i) by employing declarative trace alignment technique neglecting all the potential moves rules rather than Petri nets to represent only a partial view of in log with user actions that could never be replayed by w. the routines’ structure; (ii) by investigating sequential pattern As a consequence, this will drastically reduce the number of mining techniques [7] to examine frequent sequences of UI alignment steps required to find optimal alignments, and at the actions with common data attributes; (iii) by analyzing web log same time optimize the overall performance. Before moving mining techniques [8], which are focused on an issue similar to the next step, a new routine-based log URw is initialized. to the one of segmentation, as the input is a set of clickstreams and the goal is to extract sessions where a user engages with Trace Alignment. The second step consists of applying the a web application to fulfill a goal; (iv) by employing machine trace alignment discussed in Section II-A for any interaction learning techniques to automatically identify routine traces model w ∈ Wset and its associated model-based filtered UI without any previous knowledge of the routines’ structure. log Uφw . This enables to extract from Uφw all those user actions Finally, we are going to perform a robust evaluation of that match a distinguishable pattern with w in the form of an the proposed technique against synthetic and real-world case optimal alignment γ opt . Trace alignment allows to pinpoint studies with heterogeneous UI logs. It is worth to notice that the synchronous moves between Uφw and w. If they exist, the for the computation of the trace alignment, we will rely on user actions involved in synchronous moves are extracted and opt the highly-scalable and performing planning-based alignment stored into γsm . Note that focusing just on synchronous moves techniques implemented in [5], [9], which we can customize allows us to automatically exclude all redundant user actions for our purposes. For this reason, our main target will be to from the analysis. Then: analyze the reliability and accuracy of our technique. 1) a trace τsm consisting of the user actions associated with the synchronous moves stored in γsm opt is created; R EFERENCES w 2) a UI log Usm containing only τsm , which is required to [1] A. Bosco, A. Augusto, M. Dumas, M. La Rosa, and G. Fortino, “Discov- properly run (again) trace alignment is created; ering Automatable Routines From User Interaction Logs,” in BPM’19, w Forum track, 2019. 3) a new alignment between Usm and w with the goal to [2] S. Agostinelli, A. Marrella, and M. Mecella, “Research Challenges for compute the fitness value is performed. Intelligent Robotic Process Automation,” in BPM’19 Workshops, 2019. [3] V. Leno, A. Polyvyanyy, M. Dumas, M. La Rosa, and F. M. Maggi, w In case the fitness value is equal to 1, this means that Usm “Robotic Process Mining: Vision and Challenges,” BISE, 2020. (and, consequently, τsm ) can be replayed from the start to the [4] A. Adriansyah, N. Sidorova, and B. F. van Dongen, “Cost-Based Fitness in Conformance Checking,” in ACSD’11. IEEE, 2011. final marking of w, making τsm a valid routine trace of w. In [5] M. de Leoni and A. Marrella, “Aligning Real Process Executions and such a case, τsm is stored into URw and all the events associated Prescriptive Process Models through Automated Planning,” Expert System to the synchronous moves in τsm are removed by Uφw . On the with Application, vol. 82, pp. 162–183, 2017. [6] F. Mannhardt, M. de Leoni, H. A. Reijers, W. M. van der Aalst, and P. J. contrary, a fitness value lower than 1 indicates the presence of Toussaint, “Guided process discovery – a pattern-based approach,” Inf. at least one move in the model in τsm with respect to w, i.e., Syst., vol. 76, 2018. τsm can not be completely replayed by w and is not a valid [7] G. Dong, Sequence Data Mining. Springer-Verlag, 2009. [8] Web Usage Mining. Springer Berlin Heidelberg, 2007, pp. 449–483. routine trace, meaning that we can discard it. [9] G. De Giacomo, F. M. Maggi, A. Marrella, and F. Patrizi, “On the opt The above two steps can be repeated until γsm is not Disruptive Effectiveness of Automated Planning for LTLf -Based Trace empty, i.e., until there are synchronous moves in the computed Alignment,” in AAAI’17, 2017.