Identifying adequate back-ofice tasks to automate is a significant problem in adopting robotic process automation. Task mining on user interface logs enables departments to detect the underlying task model, but creating these logs is time and resource-consuming. Furthermore, contrary to process mining logs that can be extracted in an ex-post fashion from ERP systems, user interface logs require definitions prior to the recording: abstract activities and mapping to lower-level user interface interactions need to be specified beforehand. To this end, the presented research project proposes a new framework for desktop activity mining that enables interaction recording prior to defining tasks and activities to be mined.
Through the advent of Robotic Process Automation, the boundaries of
automationcapable business processes have shifted [
While visual modelling tools enable departments to build custom RPA bots, the technical feasibility assessment poses a significant challenge. The dificulty arises from a lack of experience and technical background compared to dedicated automation experts in IT departments. On the other hand, automation experts do not know the department’s daily tasks and, therefore, can not propose feasible tasks by themselves. Ultimately, tasks to automate can only be found through intensive exchange between both departments, which requires time and resources.
The research field around Robotic Process Automation recognises this problem of
identifying adequate tasks for automation. From the managerial perspective, researchers
propose quantified measurements to compare potential candidates in terms of
automationcapability and profitability [
Unfortunately, much information about the task executions (or cases) is not directly
available in real live scenarios [
Another possibility is recording user interface interactions and manually enriching
the log. For example, in [
The approach in [
In conclusion, the automatic identification and segmentation of task executions from
real-life logs remain one of the leading research challenges [
Therefore, this research project proposes a new approach to recording user interaction events in advance without the knowledge of any potential task candidate. This way, an employee can record all interactions of his/her daily tasks into one log. This log can then be used afterwards to mine models of tasks contained within the log. This is diferent to current works, where each log is recorded explicitly to mine one specific task model.
This project builds upon the work of [
The first part of this project will include recognising diferent instantiations of UI elements and data objects. Next, these objects are mapped to more abstract classes of objects by utilising properties provided by the MS-UIA, like the UI element name or the unique AutomationID.
As an extension to the reference model for UI logs presented in [11], this work provides semantically enriched relations between interaction events and UI objects, such as creates, requires, changes, destroys. The semantics will create precedence order relations between events independent of the user’s fixed but arbitrary chronological order, thereby revealing the half order prescribed by the application. This half-order over events simplifies the detection of recurrent patterns in the log since the relations can be interpreted as causal connections.
The next part is concerned with good algorithms for log clipping. log clipping is a new step in task mining projects concerned with the proper outline of events relevant to the task of interest. The author proposes using minimal domain knowledge in an iterative procedure to dissociate events. The algorithm leverages that events of one task are likely to form a connected component with respect to the causal relations mentioned above. Accordingly, the domain knowledge needs only to specify one distinctive event of the task. Then, a traversal through the causal relationships provides the remaining relevant events.
The final model discovery will apply state-of-the-art object-centric DFG algorithms to extract the underlying task model [12]. Developing a specialised mining algorithm for the new type of UI log is not part of this project.
The solution approach mentioned above can be broken down into three main research questions: • How can the information provided by the user interface be leveraged to identify UI element objects and data objects • What information can be exploited to automatically map instantiations of UI element objects to more abstract object classes • How can user interactions be correlated to the objects they create, require, change or destroy
An evaluation measures the UI log generation’s feasibility, efectiveness and robustness.
This includes a quantitative assessment of precision and recall of interactions against
hand-created task models. During prototyping, the author will generate logs, replaying
commonly used automation tasks from the existing literature (like transferring data from
a spreadsheet to a web form [
For robustness in real-life scenarios, a case study within a medium-sized company should confirm the results from the continuous evaluation on self-created logs. This final assessment could be enriched with a qualitative assessment of the usability, understandability and efectiveness of the models/logs from the employees through semi-structured interviews.
Preliminary prototype results reveal that the MS-UIA can be queried to extract relevant information about the UI elements. A temporal analysis of the UI state changes with the respective human interactions shows which of these are responsible for creating and deleting other UI elements. For a test data set, the approach has shown to be highly resilient to noise from pop-ups, context changes or task changes since these relate to other objects.
The next work package will target the detection of data objects via common input
values or clipboard usage. These objects will connect events from diferent applications
within the log. After this, the test data can be enlarged by replaying tasks used in current
literature that is mostly comprised of two applications [
Informaticae 175 (2020) 1–40. doi:10.3233/FI-2020-1946. [13] S. Agostinelli, F. Leotta, A. Marrella, Interactive Segmentation of User Interface Logs, in: H. Hacid, O. Kao, M. Mecella, N. Moha, H.-y. Paik (Eds.), Service-Oriented Computing, Lecture Notes in Computer Science, Springer International Publishing, Cham, 2021, pp. 65–80. doi:10.1007/978-3-030-91431-8_5.