The concept of Robotic Process Automation (RPA) has gained relevant attention in both industry and academia. RPA raises a way of automating mundane and repetitive human tasks requiring less intrusiveness with the IT infrastructure. Besides traditional user interviews and process document analysis, a common practice starts by observing the behavior of humans with the information systems while they perform the process to be automated. This sequence of human interactions with the user interface (i.e., mouse clicks and keystrokes) is stored in logs for later analysis. Analyzing these interactions brings significant benefits when conducting RPA projects. Nonetheless, some decision-based behaviors of humans require additional information to be explained. For example, a human may reject an invoice because some field is missing on a form. However, there is no interaction with that field since such information is not stored in the log. Therefore, this Ph.D. elaborates on a method to obtain additional information based on screenshots collected during the process execution. Some features are extracted from the screenshots to enrich the log, which is later used for classifying human decisions in a machine-and-human-readable form. The proposed method can be applied to generate advanced support in the RPA projects, e.g., producing an enhanced process analysis, supporting the robot development, or generating predictions and simulations. The approach has been validated using synthetic data where promising results were obtained.
In the last decade, the industry has embraced Robotic Process Automation (RPA) as a new process automation level that focuses on tackling structured and repetitive tasks quickly and eficiently. Thus, a digital workforce is enabled to mimic human employees’ behavior. This approach sharply contrasts with other paradigms for process automation that consists of the orchestration of application programming interfaces (APIs) of the software [1]. In turn, RPA implies a lower level of intrusiveness since this type of software sits on top of the information technology infrastructure of a company instead of being part of such infrastructure [2, 3]. It is acknowledged that a successful RPA adoption goes 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings htp:/ceur-ws.org CEUR IS N1613-073 beyond just simple cost savings but also contributes to improvements in terms of agility and quality [4, 5, 6].
Most RPA projects start by observing human workers performing the process, which is later automated. More precisely, terms like Robotic Process Mining [7], Task Mining [8], or Desktop Activity Mining [9] have been coined by the RPA community to exploit the UI logs, i.e., a series of timestamped event logs (e.g., mouse clicks and keystrokes) obtained by monitoring user interfaces. These methods are very convenient in helping analysts to identify candidate processes to robotize, their diferent variants, and their decision points eficiently [ 10]. However, a traditional user interface log is limited to explaining all human behavior, e.g., a decision may be motivated by a form field even though it is never directly interacted with. Therefore, human behaviors (i.e., decision points) are unexplainable by current proposals.
This problem is accentuated in the context of Business Process Outsourcing (BPO), where the processes being executed are hosted on external systems. Connections to these systems are typically made via secure connections through virtualized environments (e.g., Citrix or TeamViewer). These types of connections only allow collecting raw images of the monitored screen, i.e., screenshots, rather than the structure of the information being processed (e.g., the DOM tree of a website). This context needs some support to manage screenshots in the lifecycle that existing proposals do not cover.
Therefore, this Ph.D. intends to address these challenges based on the following
premises: (
As the research question is very generic, here we specify it in 4 sub-questions: RQ1: Are the images displayed on the screen relevant to the analysis of processes with
Robotic Process Mining (RPM)? RQ2: What alternatives exist to incorporate screen information into RPM? RQ3: How can screen information be exploited in the early stages of the RPA life cycle? RQ4: What efects would it have on the analysis and further stages of the RPA life cycle?
In answering these questions, this project proposal is planned following the Design Science methodology [11] and is organized into five main phases (P) subdivided into tasks (T). The methodology proposes that 3 of the 5 phases should be covered extensively (i.e., in-depth).
• P1. Explicate Problem: A validation is proposed to answer the RQ1, verifying that the problem is significant for the community and, therefore, an interesting contribution considering the needs of the scientific community and the industry. A Delphi [12] study is proposed for this phase. • P2. Define Requirements (in-depth): Definition of solution requirements, related to RQ3. (T.2.1) Solution requirements that encompass process discovery in image environments (i.e., processing, cleaning, and feature extraction based on a set of screenshots obtained from human monitoring). (T.2.2) The requirements of a solution that receives as input the output of the previous subtask, to be able to explain the decision points in a machine-and-human-readable form. • P3. Design and Develop Artefact (in-depth): Design and development of what is defined in P2. Which is related to RQ2 and RQ3. (T.3.1) The architecture, algorithms, and technologies to be used to address the phases of the proposed method (cf. Section 3) will be defined. This involves studying tools and algorithms such as ProM, Disco (for process discovery), Canny, Sobel, Scikit-Image, Keras, Keras-OCR1, Scikit-learn, PyTorch (for image processing), RPA-Logger 2, Spyrix or Spytech (for user behavior monitoring). (T.3.2) Implementation of the solution designed in T.3.1. • P4. Demonstrate Artefact (in-depth): Demonstration of the artifact developed in P3 taking as reference the protocol defined in [ 13], widely applied in software engineering. Two experimentations are intended to be developed. (T4.1) Using synthetic data that allows to refine the artifact. (T4.2) Using real data that allows us to bring the proposal as close as possible to a final prototype. That is trying to support a solution to RQ3 and RQ4. • P5. Evaluate Artefact: Validating the application of the proposal deployed in a real industrial context and analyzing the feedback from the users. The use case will be designed to be compatible with use in a BPO environment, which is the clearest example of the use of virtualized systems. That finally completes the answer to RQ3 and RQ4.
In this section, a method to enable advanced RPA support is described (cf. Fig. 1). This method proposes an image-based decision model discovery system for virtualized environments that ofers RPA support. At a glance, the most representative phases of the approach are: 1. Behavior monitoring to obtain a UI log. This UI log should include a screenshot for each event, e.g., using a tool such as [14]. This phase is already being extensively covered in previous investigations [14, 15]. However, the current research would require an adaptation to capture more sources of information, e.g., images. 2. Discover processes from the UI log to build the process model that best represents 1https://github.com/faustomorales/keras-ocr 2https://gitlab.com/ajramirez/rpa-logger Behaviour Monitoring
Process
Discovery UI Log
Captures
Decision point
Feature extraction
from
captures
the captured human behavior, e.g., using [10]. The phase makes explicit decision
points but lacks further information regarding how a decision is made.
Similar to the previous phase, the current state-of-the-art already provides suitable
mechanisms to conduct the discovery phase [7, 10]. However, they are required to
be adapted according to the extensions being performed in phase 1.
3. Feature extraction to transform the screenshots into objective and actionable
knowledge, e.g., the presence of specific buttons or text. These features are
automatically included as attributes (i.e., columns) in the events of the UI log. For
this extraction, several proposals exist, such as screen scraping algorithms [16] or
AI techniques, e.g., Keras-OCR. Primarily, this proposal will focus on applying
neural networks following the approach of [17, 18]. We assume that this feature
extraction will greatly increase the horizontal size of the UI log since a large number
of additional columns will be added. Therefore, it is expected that a noise reduction
technique will be necessary to discern relevant information on the screen from that
which is superficial. For example, analyzing the UI designs (i.e., how the UI is
constructed), the user attention (i.e., which parts of the UI are relevant for the
user), or the user behavior (i.e., how the user interacts and navigates through the
UIs).
4. Discovering decision models from the log enriched with the extracted features into
a machine-and-human-readable form. The discovering process is addressed for each
decision point of the process model. Herein, the extended UI log is transformed
into a dataset, which is prepared to train an explainable classifier such as decision
trees. Motivated by the work of [19], what is applied to traditional logs, the UI log
will be converted to a dataset. To do this, each case in the UI log will generate
a line in the dataset that will be labeled with the decision that is made at the
decision point.
5. Enhance discovered process by incorporating new information into the process
model. This requires the development of a new process modeling language for RPA,
or the extension of an existing one, with two main objectives: (
Although the proposed method and the application of the diferent techniques together represent a novelty at the research level, there are existing works related to each specific phase of this proposal. In the case of behavioral monitoring, there are several industrial keylogger solutions to monitor human behavior [22, 23, 24, 25]. However, they only store keystrokes and mouse clicks, in contrast to [14] keylogger that additionally extracts screenshots. In the field of image feature extraction, some existing proposals allow to identify and classify GUI (Graphical User Interface) components within a screenshot [26, 17]. GUI components are atomic graphical elements with predefined functionality, which are displayed within the GUI of a software application [17]. In this Ph.D. specific knowledge of these areas is applied to obtain enriched logs from processes to be automated.
Focusing on process discovery proposals related to this work, Agostinelli et al. [20] and Leno et al. [7] cover the complete RPA lifecycle from event capture to automatic generation of scripts for process automation and monitoring. Their way to capture data is based on an Action Logger, which captures only parts of the activity on the system through plugins. Thus, although they are focused on keyboard and mouse events, they also capture the DOM tree on events captured through the web browser. Unlike these approaches, the present work focuses on virtualized environments, where screenshots are the main source of information and there is no access to deeper elements such as the DOM tree. Furthermore, it focuses on the early stages of the RPA lifecycle since it is hypothesized that the more efort put into those stages, the better results will be obtained in subsequent ones.
Considering decision model discovery proposals, Rozinat and van der Aalst [19] use decision trees to analyze the choices made in terms of data dependencies afecting the routing of a case. However, this approach does not ofer the possibility to show graphically to a non-expert user why a decision has been made. Moreover, this solution has not been validated in RPA contexts. Furthermore, Leno et al. [27] present an algorithm that generates ”association rules” between the events that occurred and the results or decisions obtained. Nonetheless, the method of capturing information is based on a plugin, similarly to the aforementioned Action Logger which does not capture the information that the user generates outside the context of the plugin. In contrast, the present work relies on capturing the complete activity in the user interface. Thus, all interaction performed by the user is recorded to support the process discovery phase.
This research contributes to BPM research by providing an entirely image-based approach, which provides a new source of information for the study of business processes. This consists of a more efective and comprehensive discovery of human behavior based on the extraction of features from the screenshots to enrich the UI log to be analyzed. Previously, there were decision points that were not discovered or whose reasons were wrong discovered, resulting in erroneous implementations. The latter is mitigated by applying this approach, which increases the capabilities of the analysis phase and, thus, the subsequent phases of the RPA lifecycle. Besides that, it also contributes in areas such as process mining or decision model discovery, where its application is immediate. Subsequently, this approach increases the current degree of automation, so that automatable processes that were previously not automatically discoverable, now they are.
In addition, some other areas benefit from this approach such as (
Existing results related to this approach acknowledge its suitability for supporting the
RPA lifecycle. Specifically, in [ 10] a method is proposed to support the analysis of human
behavior in scenarios that highly depends on screen captures. Herein, an algorithm is
proposed to (
The next identified challenges are: (
Lastly, these challenges aim to fully automate the RPA lifecycle using new sources of information like images [30]. This final goal is ambitious and will require a gradual increase in the organization’s digital maturity, so until the point of total automation is reached, it will be necessary to consider the paradigm of human-in-the-loop [31] so that automatic techniques and human intervention coexist.
This research is part of the project PID2019-105455GB-C31 funded by MCIN/AEI/ 10.13039/501100011033. The author of this work is currently supported by the FPU scholarship program, granted by the Spanish Ministry of Education and Vocational Training (FPU20/05984) and by his Ph.D. supervisors, Andrés Jiménez Ramírez and José González Enríquez. [10] A. Jimenez-Ramirez, H. A. Reijers, I. Barba, C. Del Valle, A method to improve the early stages of the robotic process automation lifecycle, in: International Conference on Advanced Information Systems Engineering, Springer, 2019, pp. 446–461. [11] P. Johanesson, E. Perjons, Design science, An introduction to Design Science.
Springer 10 (2014) 978–1. [12] N. C. Dalkey, The Delphi method: An experimental study of group opinion, Technical
Report, RAND CORP SANTA MONICA CA, 1969. [13] P. Brereton, B. Kitchenham, D. Budgen, Z. Li, Using a protocol template for case study planning, in: 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) 12, 2008, pp. 1–8. [14] J. M. López-Carnicer, C. del Valle, J. G. Enríquez, Towards an opensource logger for the analysis of rpa projects, in: International Conference on Business Process Management, Springer, 2020, pp. 176–184. [15] V. Leno, A. Polyvyanyy, M. La Rosa, M. Dumas, F. M. Maggi, Action logger enabling process mining for robotic process automation, in: Proceedings of the Dissertation Award, Doctoral Consortium, and Demonstration Track at 17th International Conference on Business Process Management,(BPM 19), Vienna, Austria, 2019, pp. 124–128. [16] J. Bisbal, D. Lawless, B. Wu, J. Grimson, Legacy information systems: Issues and directions, IEEE software 16 (1999) 103–111. [17] K. Moran, C. Bernal-Cárdenas, M. Curcio, R. Bonett, D. Poshyvanyk, Machine learning-based prototyping of graphical user interfaces for mobile apps, IEEE Transactions on Software Engineering 46 (2018) 196–221. [18] Z. Feng, J. Fang, B. Cai, Y. Zhang, Guis2code: A computer vision tool to generate code automatically from graphical user interface sketches, in: International Conference on Artificial Neural Networks, Springer, 2021, pp. 53–65. [19] A. Rozinat, W. M. van der Aalst, Decision mining in prom, in: International conference on business process management, Springer, 2006, pp. 420–425. [20] S. Agostinelli, M. Lupia, A. Marrella, M. Mecella, Automated generation of executable rpa scripts from user interface logs, in: International Conference on Business Process Management, Springer, 2020, pp. 116–131. [21] A. Jiménez-Ramírez, J. Chacón-Montero, T. Wojdynsky, J. Gonzalez Enriquez, Automated testing in robotic process automation projects, Journal of Software: Evolution and Process (2020) e2259. [22] Spyrix Inc, Spyrix. parental & employees monitoring software, Available at www.spyrix.com, Last accesed May 2022. [23] Bestxsoftware, Best free keylogger, Available at bestxsoftware.com/es, Last accesed
May 2022. [24] Spytech Software and Design, Inc, Spytech, providing computer monitoring solutions since 1998, Available at www.spytech-web.com/spyagent.shtml, Last accesed May 2022. [25] Randhawa, A., Blackcat keylogger, Available at https://github.coma/jayrandhawa/
Keylogger, Last accesed May 2022. [26] Z. Xu, X. Baojie, W. Guoxin, Canny edge detection based on open cv, in: 2017 13th IEEE international conference on electronic measurement & instruments (ICEMI), IEEE, 2017, pp. 53–56. [27] V. Leno, A. Augusto, M. Dumas, M. La Rosa, F. M. Maggi, A. Polyvyanyy, Identifying candidate routines for robotic process automation from unsegmented ui logs, in: 2020 2nd International Conference on Process Mining (ICPM), IEEE, 2020, pp. 153–160. [28] A. Jiménez-Ramírez, I. Barba, B. Weber, C. Del Valle, Automatic generation of questionnaires for supporting users during the execution of declarative business process models, in: Business Information Systems, Springer International Publishing, Cham, 2014, pp. 146–158. [29] A. Martínez-Rojas, A. Jimenez Ramirez, J. Gonzalez Enríquez, H. Reijers, Analysing variable human actions for robotic process automation, in: International Conference on Business Process Management,(BPM 22), 2022. (In press). [30] A. Jiménez-Ramírez, Humans, processes and robots: a journey to hyperautomation, in: International Conference on Business Process Management, Springer, 2021, pp. 3–6. [31] R. C. Ruiz, A. J. Ramírez, M. J. E. Cuaresma, J. G. Enríquez, Hybridizing humans and robots: An rpa horizon envisaged from the trenches, Computers in Industry 138 (2022) 103615.