This paper presents UBGI, an innovative tool designed to enhance Robotic Process Automation (RPA) by integrating eye tracking data with user interface (UI) logs. UBGI processes and combines gaze logs with UI logs to create enriched User Behaviour (UB) logs, enabling more precise identification of user focus areas. By applying filtering masks to screenshots, UBGI highlights relevant data, facilitating analysis of user interactions. This tool enables further analysis of user behaviour through an external source, specifically eye tracking data.
In the current era of Hyperautomation [
As the first condition for automating a process, it is necessary to understand how it is
performed and even discover what process needs to be automated. To address this, the so-called
mining techniques emerged. Mining techniques like Robotic Process Mining (RPM) [
These techniques rely on the UI log [
Nevertheless, a challenge arises because of the extensive amount of data contained in the
screenshots. These screenshots capture relevant and irrelevant information, encompassing all UI
elements displayed on the screen [
This article introduces User Behaviour and Gaze Integrator (UBGI)1, a tool capable of
performing the following functions:
• Proccess and combine the eye tracking data extracted from the eye tracking software as a
gaze log with the UI log defined in. The gaze log is any CSV file that includes gaze events
as defined in [
The UB log and the generated screenshots can be used as seen in [
The remainder of this work is structured as follows. Section 2 describes the main features of the tool. In Section 3, the maturity of the tool is discussed. Section 4 provides links to the demonstration of the tool in the form of a video. Finally, the conclusions of this work are presented in Section 5
The proposed tool has been developed as a web application using Django REST framework for Python. JavaScript is used for client-side rendering, and PostgreSQL is used as the database.
The innovative features of the tool reside in two phases: the UB Log Gaze Enricher phase and
the Filtering Mask phase. Before these two phases, the user must record the execution of the
process, i.e., the case study. The case study consists of one or more UI logs and gaze logs for
one or more scenarios. To obtain the UI log for each scenario with the screenshots associated
with each action, loggers are used to monitor the user’s activity and save it. To obtain the gaze
log, it is necessary to have an eye tracking tool, such as webcam-based eye tracking software
that can capture the POGs on the screen during the case study recording. WebgaWzer.js is an
open source webcam-based eye tracking web application 2 [
Once the case study is available and loaded into the UBGI tool, it is executed through the
following phases:
• UB Log Gaze Enricher phase: In this phase, we integrate the UI log with the gaze log to
form the UB log. The eye tracking software data is analyzed to identify gaze events such
as fixations, saccades, and dispersion values [
The UB log generated in the UB Log Gaze Enricher phase and the screenshots that include the filtering mask can be downloaded as a.zip file once the case study is executed.
The UBGI tool is currently in a stable version that meets its purposes. Nevertheless, there are several lines of research and future work that can contribute to a higher level of maturity.
First, the UBGI tool demonstrates flexibility in adjusting diferent parameters and formats of UI logs. However, it does not have the same level of adaptability for gaze logs. In future work,
it is necessary to deepen the understanding of the various available eye tracking software and apply a similar treatment to that used for UI logs. This will allow UBGI to process any gaze log from any eye tracking software.
Second, determining the attention areas and gaze masks in the Filtering mask phase only considers fixation and dispersion gaze events. Other parameters can be configured to determine an optimal drawing of the attention areas and, conversely, the filtering mask (e.g., thresholds, saccade eye movements, etc.).
Third, the artefacts generated in the phases of the UBGI tool contain important information only if they are going to be processed. Therefore, it is important to integrate the artefacts generated in the UBGI phases into new phases or modules that can be created within the tool for the discovery of automatable processes. Even include the UBGI tool in other tools that perform the function of extracting features from the UI for process discovery.
UBGI has been subjected to pilot tests in simulated environments that replicate real-use conditions. These tests have provided valuable data to identify possible improvements and confirm the efectiveness of UBGI in data integration and analysis. However, implementation in real productive environments will be crucial to evaluate its performance in intensive use situations and its capacity to handle massive amounts of data.
In this section, we present a demonstration of the artefacts generated by the UBGI tool. To do this, we will record and execute a complete case study3. The case study simulates a process within the HR department of a company. Specifically, the case study involves an administrative worker who must attach the COVID 19 certificates of company employees to the database from a folder. If an employee is not found in the database, the administrator must report the missing employee with his COVID 19 certificate.
For recording the UI log, we use Steprecorders, a native application of the Windows operating system. To record the gaze log, we use the Webgazer.js eye tracking software.
The User Behaviour Gaze Integrator (UBGI) tool contributes to the BPM community with the ability to integrate and analyse eye tracking data and user interface logs, enabling the consideration of broader contextual information when studying user behaviour. Therefore, this opens up new lines of work in fields such as RPA, UBM, and RPM.
The proposed tool processes user activities by first integrating UI logs with gaze logs to form an enriched UB log. This is achieved through the UB Log Gaze Enricher phase, which matches user actions with corresponding gaze events using timestamps. In the subsequent Filtering Mask phase, the tool generates modified screenshots highlighting attention areas by overlapping a gaze-based mask, revealing relevant and irrelevant information found in the screenshots, whether the user is performing an action or not. These capabilities allow for a
more detailed analysis of user interactions, providing valuable insights to further analysis for process automation strategies.
This research was supported by the EQUAVEL project PID2022-137646OB-C31, funded by MICIU/AEI/10.13039/501100011033 and by FEDER, UE; the DISCOVERY project (2021/C005/00148631), funded by Unión Europea NextGeneration EU and “Plan de Recuperación, Transformación y Resiliencia” of the Ministry of Economic and Digital Transformation; and the grant FPU20/05984 funded by MICIU/AEI/10.13039/501100011033 and by FSE+.