This paper presents a tool, called Action Logger, for recording user interface (UI) logs, i.e., logs of user interactions with information systems. By generating output suitable for process mining, the tool aims to introduce process mining methods, techniques, and tools for supporting Robotic Process Automation (RPA) activities, e.g., robot discovery and implementation. Action Logger o ers unique capabilities, including logging relevant user actions at a granularity level suitable for RPA, data-awareness, and context-independence.
Automating repetitive clerical tasks is a common approach to enhance e ciency
and reduce defect rates in business processes. Robotic Process Automation
(RPA) [
The creation of a bot requires an in-depth knowledge of the tasks to be
automated, IT systems involved and their UIs, as well as how users interact with
them. In current practice, this knowledge is gained via interviews and workshops
with the stakeholders, and analysis of unstructured data, e.g., video recordings of
users working with the systems [
Process Mining [
The adoption of process mining techniques for RPA is hampered by the absence of tools capable of recording UI logs that (i) can be given as input to process mining tools and (ii) contain information at the granularity level suitable for RPA. This paper presents a tool, namely Action Logger, that addresses this gap. The paper discusses the requirements addressed by Action Logger, the tool's architecture, its development status and directions for enhancement. The paper is complemented by a screencast3, a tutorial4 and the tool itself.5 2
This section discusses the requirements that guided the development of UI Logger, its architecture, and a motivating example. 2.1
To generate UI logs amenable for further RPA-related analysis with process mining, a logger tool should ensure these functional requirements: RQ1: Relevance. A tool should only record meaningful, value-adding actions. For example, actions of moving the mouse or clicking on the background of a web site should not be recorded as they do not impact the outcome of a task. However, button clicks and URI link clicks are essential actions and, therefore, should be captured.
RQ2: Granularity. A tool should record actions at a level of details su cient to fully reconstruct the performed task. For example, the logger should be able to di erentiate between di erent types of mouse clicks, e.g., clicking on a button versus clicking a link in a web browser.
RQ3: Data-awareness. In addition to performed actions, a tool should record the data that supports them. These data is crucial in discovering the rule-based decision making logic of the process. The data can also be used to discover data transformations to enhance the quality of the discovered process model. To enable performance analysis, the tool should also record timestamps associated with the performed actions.
RQ4: Context-independence. A tool should record actions in a way that
they can be replayed with the same e ects on di erent machines, platforms,
under various circumstances and contexts, e.g., di erent UI layouts.
RQ5: Interoperability. A tool should record UI logs in a format supported
by process mining tools, for example CSV and/or XES [
To the best of our knowledge, no solution satis es all the above requirements. Available UI action recording tools, like WinParrot (www.winparrot.com) and JitBit Macro Recorder (www.jitbit.com/macro-recorder), record low-level actions only, e.g., clickstreams and keystrokes. The recorded actions refer to pixel 3 The screencast is published at https://youtu.be/SvPuOdWfByc. 4 The tutorial is available at https://github.com/apromore/RPA_UILogger. 5 The tool is available at https://github.com/apromore/RPA_UILogger/releases. t p m s i L o g fil t eri n g a n d acifil noi OS Clipboard Clipboard Events CSV Logger
Excel Events Action Logger
Backend applications Cell Events
Webpage Events Chrome
Events
P r o c e s s d i s c o v e r y
R o b o t i m p l e m e n t a t i o n
R o b o t i c P r o c e s s A u t o m a t i o n coordinates, e.g., click mouse at coordinates 341, 568, that depend on screen resolution and window size. Some tools, like WinParrot, save information regarding the application where the action was performed. However, they do not identify application-speci c functionality, e.g., editing or copying a cell in a spreadsheet. Most of the tools do not capture timestamps. However some of them save the delays between actions. None of the existing tools generates les in the format that can be directly consumed by process mining techniques.
RPA tools, like Automation Anywhere (www.automationanywhere.com) and UIPath (www.uipath.com), provide recording capabilities to generate executable scripts. However, the generated logs are only readable within the environment of the RPA solutions themselves. 2.2
Action Logger records user actions performed in Excel and Chrome web browser, two often used applications for o ce tasks. It includes two separate plug-ins, one for each application. The plug-ins are implemented as event listeners and send the information about performed actions as JSON objects to the logging component, which generates and updates the UI log on the y. To record the actions, the logger uses APIs of the corresponding applications. The browser actions are recorded at the level of the Document Object Model, capturing the involved web elements, e.g., text elds, buttons, and links. The tool also monitors the clipboard to record relevant actions, e.g., copying and pasting of les. (a) Student records spreadsheet (b) New Record creation form Fig. 2: Extract of spreadsheet with data that needs to be transferred to a form.
The architecture of the tool and the envisaged pipeline for employing the logger for RPA are shown in Fig. 1. The tool stores all the data values used in the context of every recorded action. For example, for an action performed in a spreadsheet, the tool captures the information about the cell, its current value, workbook, and active sheet in which the action took place. The generated logs are stored in the CSV format suitable for process mining investigations.
We also developed a log simpli cation tool to reduce the size of the UI log in a semantics-preserving manner.6 The log simpli er works in two steps. First, it converts operating system-level actions into application-level actions (e.g., a get-cell and a copy action are merged into a single copy-cell action). Next, it removes redundant actions, speci cally redundant navigation actions in Excel (user selects di erent cells and then eventually copies the value in a relevant cell), copy actions without corresponding pasting, and copy actions that are later overwritten (douple copying). The log simpli er is implemented as a set of regular expression search-and-replace rules. Some of these expressions are purely control- ow based, e.g., navigation events and double copying, while others are data-aware, e.g., double editing of a text eld. One typical task automated using RPA is the task of transferring data from one system to another, for example from a spreadsheet to a form of a web-based information system. To demonstrate Action Logger, we use an Excel spreadsheet that contains students' contact details, e.g., full name, date of birth, phone number, and email, and a web form. We manually transferred the data about the rst student in the spreadsheet from Excel into the web form, refer to Fig. 2, and recorded all the performed actions using Action Logger. Fig. 3 shows a fragment of the UI log produced by the logger; for the layout considerations, we do not show all the recorded event attributes.
To demonstrate that the recoded UI log is suitable for process mining investigations, Fig. 4 shows the directly follows graph automatically discovered from the log in Fig. 3 using Apromore (www.apromore.org) 3
To validate Action Logger in practice, we established a cooperation with the University of Melbourne, Australia. In particular, we work closely with the University Services team, which is responsible for the admission and scholarship 6 The tool is available at https://github.com/apromore/RPA_SemFilter/releases. allocation processes in the university. The team uses Action Logger to record the workers while performing routine operations and provides feedback with respect to validity (all recorded actions are correct and relevant) and completeness (all the important and relevant actions are recorded) of the recorded logs; note that many of these routines involve work with spreadsheets and web-based frontends of the university IT systems, similar to the example from Section 2.3. We use the collected feedback to improve the tool. We plan to use the recorded logs to support the tasks of robot identi cation and implementation, in which the University Services team is currently engaged.
This paper presented Action Logger, a tool for automatic recording of user interactions with IT systems when performing tasks in a business process. The UI logs recorded by Action Logger are speci cally designed to be imported into process mining tools, such as Apromore, thus enabling the analysis of repetitive routines for RPA. In its next releases, we foresee that the tool will support the recording of user interactions with a wider range of software applications (beyond Excel and browsers). We will also explore the use of image recognition techniques and OCR to record user interactions with applications that do not provide suitable APIs for detecting ne-grained events.
Acknowledgments. The authors thank Stanislav Deviatykh for his help in implementing the log simpli er. This research is funded by the Australian Research Council (DP180102839), the Estonian Research Council (IUT20-55), and the European Research Council (project PIX).