=Paper=
{{Paper
|id=Vol-3299/Paper10
|storemode=property
|title=Ex-Post Identification of Task Models With Causally Ordered User Interface Logs (Extended Abstract)
|pdfUrl=https://ceur-ws.org/Vol-3299/Paper10.pdf
|volume=Vol-3299
|authors=Dominic A. Neu
|dblpUrl=https://dblp.org/rec/conf/icpm/Neu22
}}
==Ex-Post Identification of Task Models With Causally Ordered User Interface Logs (Extended Abstract)==
https://ceur-ws.org/Vol-3299/Paper10.pdf
Ex-Post Identification of Task Models With Causally
Ordered User Interface Logs (Extended Abstract)
Dominic A. Neu1
1
Institute for Information Systems, Saarland University, Germany
Abstract
Identifying adequate back-office tasks to automate is a significant problem in adopting robotic
process automation. Task mining on user interface logs enables departments to detect the
underlying task model, but creating these logs is time and resource-consuming. Furthermore,
contrary to process mining logs that can be extracted in an ex-post fashion from ERP systems,
user interface logs require definitions prior to the recording: abstract activities and mapping
to lower-level user interface interactions need to be specified beforehand. To this end, the
presented research project proposes a new framework for desktop activity mining that enables
interaction recording prior to defining tasks and activities to be mined.
Keywords
task mining, causal logs, object-centric UI log
1. Introduction and Motivation
Through the advent of Robotic Process Automation, the boundaries of automation-
capable business processes have shifted [1]. Robotic Process Automation operates on
the user interface of enterprise software. Therefore, integrating systems through the
existing user interface eliminates the need for separate integration APIs. Furthermore, the
visual modelling environment provided by RPA providers enabled departments to start
their automation projects without the involvement of the IT department or professional
programmers. Moreover, recent advances in artificial intelligence expanded the scope of
tasks a computer can take over.
While visual modelling tools enable departments to build custom RPA bots, the
technical feasibility assessment poses a significant challenge. The difficulty arises from a
lack of experience and technical background compared to dedicated automation experts
in IT departments. On the other hand, automation experts do not know the department’s
daily tasks and, therefore, can not propose feasible tasks by themselves. Ultimately, tasks
to automate can only be found through intensive exchange between both departments,
which requires time and resources.
ICPM 2022 Doctoral Consortium and Tool Demonstration Track
email: dominic.neu@uni-saarland.de (D. A. Neu)
orcid: 0000-0002-9806-6520 (D. A. Neu)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
46
2. Problem statement and state-of-the-art
The research field around Robotic Process Automation recognises this problem of iden-
tifying adequate tasks for automation. From the managerial perspective, researchers
propose quantified measurements to compare potential candidates in terms of automation-
capability and profitability [2, 3]. Whereas the works of [4, 5, 6, 7, 8] leverage insights
from process mining to discover potential task models from user interface logs.
Unfortunately, much information about the task executions (or cases) is not directly
available in real live scenarios [6]. However, this information is necessary to calculate
the previously proposed measurements and apply the process mining algorithms. One
way to get around this limitation is to have employees estimate the values of essential
criteria, such as the number of exceptional cases or the execution time.
Another possibility is recording user interface interactions and manually enriching
the log. For example, in [4] and [8], the log is segmented implicitly into traces by the
user when he starts and stops the recording. Additionally, the user abstracts events to
activities with tool support.
The approach in [5] solves the segmentation task by building a dominator tree from the
directly-follows-graph (DFG). An integral part of the DFG approach is the automated
mapping of events to more abstract activities through additional UI information, such
as the button name or application-specific parameters. For this, the solution requires
application-specific add-ons and the log to contain only events from one task.
In conclusion, the automatic identification and segmentation of task executions from
real-life logs remain one of the leading research challenges [6, 9].
3. Proposed approach and methodology
Therefore, this research project proposes a new approach to recording user interaction
events in advance without the knowledge of any potential task candidate. This way, an
employee can record all interactions of his/her daily tasks into one log. This log can then
be used afterwards to mine models of tasks contained within the log. This is different to
current works, where each log is recorded explicitly to mine one specific task model.
This project builds upon the work of [5] to leverage additional UI element properties
for automatically mapping events to activities. However, instead of recording events
within the applications through add-ons, the author proposes using the application-
generic Microsoft User Interface Automation API (MS-UIA), which offers many insightful
properties. Furthermore, to tackle the problem of a proper case distinction, the author
proposes to transfer the object-centric approach from process mining to UI logs [10] as
user interaction events may be related to UI elements and data objects.
The first part of this project will include recognising different instantiations of UI
elements and data objects. Next, these objects are mapped to more abstract classes of
objects by utilising properties provided by the MS-UIA, like the UI element name or the
unique AutomationID.
As an extension to the reference model for UI logs presented in [11], this work provides
47
semantically enriched relations between interaction events and UI objects, such as creates,
requires, changes, destroys. The semantics will create precedence order relations between
events independent of the user’s fixed but arbitrary chronological order, thereby revealing
the half order prescribed by the application. This half-order over events simplifies the
detection of recurrent patterns in the log since the relations can be interpreted as causal
connections.
The next part is concerned with good algorithms for log clipping. log clipping is a new
step in task mining projects concerned with the proper outline of events relevant to the
task of interest. The author proposes using minimal domain knowledge in an iterative
procedure to dissociate events. The algorithm leverages that events of one task are likely
to form a connected component with respect to the causal relations mentioned above.
Accordingly, the domain knowledge needs only to specify one distinctive event of the task.
Then, a traversal through the causal relationships provides the remaining relevant events.
The final model discovery will apply state-of-the-art object-centric DFG algorithms to
extract the underlying task model [12]. Developing a specialised mining algorithm for
the new type of UI log is not part of this project.
The solution approach mentioned above can be broken down into three main research
questions:
• How can the information provided by the user interface be leveraged to identify UI
element objects and data objects
• What information can be exploited to automatically map instantiations of UI
element objects to more abstract object classes
• How can user interactions be correlated to the objects they create, require, change
or destroy
An evaluation measures the UI log generation’s feasibility, effectiveness and robustness.
This includes a quantitative assessment of precision and recall of interactions against
hand-created task models. During prototyping, the author will generate logs, replaying
commonly used automation tasks from the existing literature (like transferring data from
a spreadsheet to a web form [8]).
For robustness in real-life scenarios, a case study within a medium-sized company
should confirm the results from the continuous evaluation on self-created logs. This final
assessment could be enriched with a qualitative assessment of the usability, understand-
ability and effectiveness of the models/logs from the employees through semi-structured
interviews.
4. Preliminary Results & Future Work
Preliminary prototype results reveal that the MS-UIA can be queried to extract relevant
information about the UI elements. A temporal analysis of the UI state changes with
the respective human interactions shows which of these are responsible for creating and
deleting other UI elements. For a test data set, the approach has shown to be highly
48
resilient to noise from pop-ups, context changes or task changes since these relate to
other objects.
The next work package will target the detection of data objects via common input
values or clipboard usage. These objects will connect events from different applications
within the log. After this, the test data can be enlarged by replaying tasks used in current
literature that is mostly comprised of two applications [13, 5]. The common tasks are
then recorded in one log containing many more tasks and noisy activities. This log should
reflect the more realistic setting when the task for mining is not known at recording
time. At last, applying the two current state-of-the-art task mining tools on the realistic
log should compare the strengths and weaknesses of these approaches to the presented
approach.
References
[1] W. M. P. van der Aalst, Hybrid Intelligence: To automate or not to automate, that
is the question, Ijispm-International Journal of Information Systems and Project
Management 9 (2021) 5–20. doi:10.12821/ijispm090201.
[2] J. Wanner, A. Hofmann, M. Fischer, F. Imgrund, C. Janiesch, J. Geyer-Klingeberg,
Process selection in RPA projects-towards a quantifiable method of decision making,
in: ICIS 2019 Proceeedings, volume 6, 2019.
[3] S. Rechberger, S. Oppl, Selecting processes for RPA, in: C. Czarnecki, P. Fettke
(Eds.), Robotic Process Automation: Management, Technology and Applications,
De Gruyter, 2021, pp. 91–109. doi:10.1515/9783110676693-005.
[4] C. Linn, P. Zimmermann, D. Werth, Desktop Activity Mining - A new level of detail
in mining business processes, in: Workshop Der INFORMATIK 2018, Lecture Notes
in Informatics (LNI), Gesellschaft für Informatik, Bonn, 2018, pp. 245–258.
[5] V. Leno, A. Augusto, M. Dumas, M. La Rosa, F. M. Maggi, A. Polyvyanyy,
Identifying Candidate Routines for Robotic Process Automation from Unsegmented
UI Logs, in: 2020 2nd International Conference on Process Mining (ICPM), 2020,
pp. 153–160. doi:10.1109/ICPM49681.2020.00031.
[6] D. Choi, H. R’bigui, C. Cho, Candidate Digital Tasks Selection Methodology for
Automation with Robotic Process Automation, Sustainability 13 (2021) 8980.
[7] E. L. Klijn, F. Mannhardt, D. Fahland, Classifying and Detecting Task Executions
and Routines in Processes Using Event Graphs, in: A. Polyvyanyy, M. T. Wynn,
A. Van Looy, M. Reichert (Eds.), Business Process Management Forum, Lecture
Notes in Business Information Processing, Springer International Publishing, Cham,
2021, pp. 212–229. doi:10.1007/978-3-030-85440-9_13.
[8] S. Agostinelli, M. Lupia, A. Marrella, M. Mecella, Reactive synthesis of software
robots in RPA from user interface logs, Computers in Industry 142 (2022) 103721.
doi:10.1016/j.compind.2022.103721.
[9] J. Gao, S. J. van Zelst, X. Lu, W. M. P. van der Aalst, Automated Robotic
Process Automation: A Self-Learning Approach, in: On the Move to Mean-
ingful Internet Systems: OTM 2019 Conferences, volume 11877 of Lecture
49
Notes in Computer Science, Springer International Publishing, 2019, pp. 95–112.
doi:10.1007/978-3-030-33246-4_6.
[10] A. F. Ghahfarokhi, G. Park, A. Berti, W. M. P. van der Aalst, OCEL: A Stan-
dard for Object-Centric Event Logs, in: L. Bellatreche, M. Dumas, P. Karras,
R. Matulevičius, A. Awad, M. Weidlich, M. Ivanović, O. Hartig (Eds.), New
Trends in Database and Information Systems, Communications in Computer and
Information Science, Springer International Publishing, Cham, 2021, pp. 169–175.
doi:10.1007/978-3-030-85082-1_16.
[11] L. Abb, J.-R. Rehse, A Reference Data Model for Process-Related User Interaction
Logs, in: C. Di Ciccio, R. Dijkman, A. del Río Ortega, S. Rinderle-Ma (Eds.),
Business Process Management, Lecture Notes in Computer Science, Springer Inter-
national Publishing, Cham, 2022, pp. 57–74. doi:10.1007/978-3-031-16103-2_7.
[12] W. M. P. van der Aalst, A. Berti, Discovering Object-centric Petri Nets, Fundamenta
Informaticae 175 (2020) 1–40. doi:10.3233/FI-2020-1946.
[13] S. Agostinelli, F. Leotta, A. Marrella, Interactive Segmentation of User Interface
Logs, in: H. Hacid, O. Kao, M. Mecella, N. Moha, H.-y. Paik (Eds.), Service-Oriented
Computing, Lecture Notes in Computer Science, Springer International Publishing,
Cham, 2021, pp. 65–80. doi:10.1007/978-3-030-91431-8_5.
50