Within the field of process mining, there is an emerging topic of research regarding the feedback efects of process mining tools on behavior of process analysts. Due to its more recent emergence, not much research has been done on the topic. To address this gap, my PhD project will focus answering the question how does process automation impact the work of knowledge workers. The setting of auditing is selected for a case study that will examine the short and long term efects of process automation. My PhD research aims to discover both the positive and negative efects stemming from the implementation of process automation within the audit setting. Insights into this topic allow for better organizational decision-making regarding the implementation of process automation tools. Furthermore, it can serve as a basis for further theory building in a developing area of process mining research.
It is said that a fool with a tool is still a fool. But what about an expert with a tool? Do they become better
experts or lose their expertise altogether? New types of process mining tools are fundamentally changing
the way in which knowledge workers are approaching their work. The expectation is that various
professional services, such as auditing, will be drastically impacted by those tools’ implementation
[
Research on process mining has traditionally focused on developing or improving algorithms for
automatic process discovery, conformance checking, and process enhancement [
The research question addressed in this research project is how does process automation impact the work of knowledge workers. The project will focus on examining how knowledge workers are afected by the implementation of process automation, as well as the benefits and negative feedback efects of process automation on the knowledge worker. The goal is to develop a profound empirical understanding of how process automation afects the work of knowledge workers.
The rest of this paper will be structured as follows. Section 2 first describes related work regarding the specific knowledge work setting of process auditing, second, it describes research regarding human factors and process automation. Section 3 discusses the research objectives. In Section 4, methodologies
CEUR
ceur-ws.org are discussed for each of the associated work packages. Section 5 discusses the preliminary results that have been achieved so far.
Auditors play a pivotal role in the eficient working of financial markets by assessing the credibility and quality of corporate financial reports. To this end, they follow a structured judgment and decisionmaking process. As for most professional knowledge workers, the advancement of technology, including process automation, has created opportunities for auditors to increase the eficiency and efectiveness of their work [10]. The use of data analytics techniques to analyze large volumes of data to identify patterns, anomalies, and potential fraud in an audit setting is commonly referred to as Audit Analytics [11] and has been the subject of multiple research studies [12, 13, 14, 15].
So far, we know surprisingly little about the actual impact of process mining tools on the auditor’s analytical work. And indeed, audit research has formulated arguments why the impact of new technology has to be cautiously investigated. Arnold and Sutton (1998) developed the Theory of Technology Dominance for describing the necessary conditions under which professional knowledge workers would rely on intelligent systems, and for theorizing the probability of success or failure as a result of this reliance [16]. The theory crystallized from observations of unintended long-term consequences of the usage of technology by knowledge workers. Such negative consequences are referred to as technology dominance, automation bias, or overreliance on technology [17]. Multiple studies demonstrate the detrimental impact technology dominance can have on decision-making in an auditing context, such as increased biases [18], impeded knowledge development with novices [19], and decreased levels of declarative knowledge held by long-term users of these systems [20].
A more positive note can be found in the scant studies on knowledge transfer from knowledge-based systems to the users of those systems. Preliminary evidence exists that certain design elements, e.g. explanation types or system interfaces embedding expert knowledge structures, have a positive impact on knowledge acquisition by novices [18, 21, 22]. However, research on how the performance of auditors is impacted by tools such as process mining is largely missing.
Parasuraman et al. (2000) define automation as putting a system in place that accomplishes a function that was or could be carried out by a human [23]. They emphasize that automation is a gradual phenomenon and distinguish ten levels from no automation at all to autonomous action by the automation without human involvement. Automation can be related to diferent stages of human information processing along the four stages of acquisition, analysis, decision making, and action.
The availability of automation, can have undesired and unexpected consequences. Next to the desired and expected use, Parasuraman and Riley (1997) describe misuse, disuse, and abuse [24]. Misuse refers to over-reliance, which is likely to result in failures and biases. Disuse describes under-utilization, for instance when floods of false alarms are increasingly ignored. Abuse related to the automation being put in place without considering the complex consequences on human performance. Two negative efects are specifically important in this context: complacency and skill degradation.
Complacency relates to the level of trust a human has developed into automation in a particular setting. High trust, despite the automation not being fully reliable, may lead to over-reliance [25, 26]. In this case, the human pays not enough attention to checking the correct operation of the automation. In this way, complacency represents a short-term efect. Such efects could be relevant in the context of process analysis when analysts develop an over-reliance on the outputs of a process mining tool. Skill degradation refers to a negative efect over a longer period of time. For the context of process analysis, cognitive skill degradation is most relevant. It represents the reduction of thinking, reasoning, and decision- making skills [27].
Both of these concepts have their origin in the field of engineering psychology. In general, however, the humans operating in these settings have the same cognitive characteristics as process analysts, but their tasks are diferent. Therefore, it is not clear to which extent automation of process analysis may lead to negative side efects and which impact this might have.
As stated earlier, the goal is to develop a profound empirical understanding of the impact of process automation on the work of knowledge workers by answering the research question: how does process automation afect the work of knowledge workers. A collaboration with one of the big four accounting ifrms is set up to provide an environment in which auditors are given a process automation tool to assist them with their audit tasks. This environment will serve as the basis for a case study that aims to provide the desired empirical understanding mentioned before. The sample size of auditors in the case study is rather limited. As a result, the PhD project will primarily make use of qualitative research techniques. Based on the literature presented in the previous section, the following research objectives are proposed to answer the research question: • RO1: Determine the influence process automation use has on the work of knowledge workers • RO2: Develop a list of concrete auditor tasks • RO3: Determine the core tasks and related skills that constitute expertise in auditing • RO4: Benchmark the skills of auditors in those core tasks • RO5: Examine the short and long term efects of process automation on those tasks
For each of the research objectives proposed in the previous section, a work package is constructed that aims to achieve each objective. The following sections discuss the goals and methodologies used for each work package.
This work package has two primary goals. The first is to establish an initial model that can be used to measure what kind of efects can be expected when introducing process automation to knowledge work. The second goal is to examine the characteristics of the process automation tool being introduced at the accounting firm. The combination of these two deliverables will serve as a basis for future work packages.
To achieve the first goal, models pertaining to technology acceptance and automation efects from diferent research fields are collected and compared. These are compared for similarities, contradictions and shortcomings. Determining what drives the use of and reliance on a process automation tool, allows to better determine the scope of the later stages of the project. If the audit case indicates that reliance will be achieved at a slow rate, long term efects may not be identifiable within the scope of this PhD. Knowing the possible efects of process automation will form the basis for further research in later work packages.
The second goal of this work package is to examine the process automation tool being implemented in the accounting firm. The tool has diferent modules that each serve a diferent purpose and automate to a diferent degree. As stated in the related work section, Parasuraman (2000) proposes that automation can occur on ten diferent levels, and automation can occur to a diferent degree for each of the stages of human information processing [23]. Using semi-structured interviews and document analysis, a deep understanding is obtained for each for the diferent modules that the new automation tool ofers. For each of the modules, its activities are organized according to the stages of human information processing, and the levels of automation. Once a deep understanding of the tool is realized, this can further help shape what automation efects can be expected when compared to the model from the first goal.
There is currently no comprehensive list present in literature that fully examines all tasks an auditor performs. The goal of this work package is to create this comprehensive list. First, literature will be examined to construct a preliminary list of potential tasks that an auditor performs. This list will then be validated and enriched through document analysis and observations. The partner accounting firm will provide training documents and guidelines that are also given to new auditors. These documents can serve as a first validation for the created list, and provide additional insights more closely related to the case. Observations will then be performed by way of think-aloud protocol. This will capture additional tasks and validate or refute previously collected tasks. This think-aloud protocol will be conducted for auditors of difering levels of expertise. These observations will occur multiple times until a suficiently comprehensive list of tasks is obtained.
Starting from the comprehensive list of tasks, the goal of this work package is to determine the core of what makes an expert auditor. Research on knowledge work shows that, for auditing, one of the primary measures for expertise is pattern recognition [28, 29]. Those tasks that are central to auditing expertise will most likely heavily involve pattern recognition. To validate this and to identify which of the tasks in the list make up the core of auditing expertise, semi-structured interviews are conducted with audit experts using the ACTA protocol as a guideline [30]. This protocol describes an approach to interviews aimed at capturing and analyzing knowledge regarding applied cognitive tasks.
To examine the efects of process automation tools, a basis for comparison must be established. The goal of this work package is to benchmark the skills of auditors using an observational study. The goal will be to measure their skills in performing the tasks that were previously established in work package 3. Due to the small sample size of this case, all available auditors will be benchmarked. These auditors vary in level of expertise. This is desirable, as literature states that novice and expert knowledge workers are afected diferently by process automation. As a result, a richer spectrum of results can be examined.
Once the process automation tool has been implemented, its efects can be examined over time. The current goal is to determine the beneficial en detrimental impact the process automation had. Expected beneficial efects include decision performance and skill improvement. Negative efects are expected in the form of both complacency in the short term and skill degradation in the long term. Due to the small sample size of auditors in this case, the same auditors that were benchmarked in work package 4 can once again be examined with an observational study for the same skills.
Using the results from this work package, a comparative analysis is made with the model constructed in work package 1. The final result of this work package will be a validated model of the impacts of process automation tools on knowledge workers for the case of auditing.
All preliminary results pertain to work package one. For the first goal of work package one, the model shown in Figure 1 is established that combines perspectives from the diferent research fields of engineering psychology, information systems, and accounting. This model will serve as a basis to determine what drives knowledge workers to use process automation tools. This can be used to determine the speed of adoption of the process automation tool once implemented. Additionally, this model serves to provide insights into the types of efects that can be expected from the implementation of a process automation tool.
The model consists of three phases. The first phase shows diferent characteristics that drive reliance on a process automation tool. As reliance increases, the second phase of the models describes possible efects that start occurring as a result of efective use and reliance. Short term efects of automation have shown to be beneficial, while efects on skills related to the automated task have shown to be subject to complacency. A relatively underexplored area regarding the topic of process automation efects, is that of indirectly afected skills. These are skills not related to the task being automated. While some research has shown that these skills can improve [31], other research has shown the contrary [32]. Phase three shows the long term efects of process automation on skills. Skills that have sufered from complacency, will start to erode. Skills that have been empowered are theorized to improve over time.
The process automation tool from the accounting firm itself has also been examined on a high level to determine the diferent levels of automation that are provided to auditors throughout the auditing process.
The author(s) have not employed any Generative AI tools. [3] J. Eggers, A. Hein, M. Böhm, H. Krcmar, No Longer Out of Sight, No Longer Out of Mind? How Organizations Engage with Process Mining-Induced Transparency to Achieve Increased Process Awareness, Business & Information Systems Engineering 63 (2021) 491–510. URL: https: //doi.org/10.1007/s12599-021-00715-x. doi:10.1007/s12599- 021- 00715- x. [4] P. Badakhshan, B. Wurm, T. Grisold, J. Geyer-Klingeberg, J. Mendling, J. vom Brocke, Creating business value with process mining, The Journal of Strategic Information Systems 31 (2022) 101745. URL: https://www.sciencedirect.com/science/article/pii/S0963868722000415. doi:10.1016/j.jsis. 2022.101745. [5] L. Zimmermann, F. Zerbato, B. Weber, Process Mining Challenges Perceived by Analysts: An Interview Study, in: A. Augusto, A. Gill, D. Bork, S. Nurcan, I. Reinhartz-Berger, R. Schmidt (Eds.), Enterprise, Business-Process and Information Systems Modeling, Springer International Publishing, Cham, 2022, pp. 3–17. doi:10.1007/978- 3- 031- 07475- 2_1. [6] F. Zerbato, P. Sofer, B. Weber, Process Mining Practices: Evidence from Interviews, in: C. Di Ciccio, R. Dijkman, A. del Río Ortega, S. Rinderle-Ma (Eds.), Business Process Management, Springer International Publishing, Cham, 2022, pp. 268–285. doi:10.1007/978- 3- 031- 16103- 2_19. [7] V. Venkatesh, J. Thong, X. Xu, Unified Theory of Acceptance and Use of Technology: A Synthesis and the Road Ahead, Journal of the Association for Information Systems 17 (2016). URL: https: //aisel.aisnet.org/jais/vol17/iss5/1. doi:10.17705/1jais.00428. [8] D. L. Goodhue, R. L. Thompson, Task-Technology Fit and Individual Performance, MIS Quarterly 19 (1995) 213–236. URL: https://www.jstor.org/stable/249689. doi:10.2307/249689, publisher: Management Information Systems Research Center, University of Minnesota. [9] J. vom Brocke, M. Jans, J. Mendling, H. A. Reijers, A Five-Level Framework for Research on Process Mining, Business & Information Systems Engineering 63 (2021) 483–490. URL: https: //doi.org/10.1007/s12599-021-00718-8. doi:10.1007/s12599- 021- 00718- 8. [10] M. Eulerich, A. Masli, J. Pickerd, D. A. Wood, The Impact of Audit Technology on Audit Task Outcomes: Evidence for Technology-Based Audit Techniques, Contemporary Accounting Research 40 (2023) 981–1012. URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/1911-3846.12847. doi:10.1111/1911- 3846.12847, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/19113846.12847. [11] H. Li, J. Dai, T. Gershberg, M. A. Vasarhelyi, Understanding usage and value of audit analytics for internal auditors: An organizational approach, International Journal of Accounting Information Systems 28 (2018) 59–76. URL: https://www.sciencedirect.com/science/article/pii/S1467089516301415. doi:10.1016/j.accinf.2017.12.005. [12] H. Brown-Liburd, M. A. Vasarhelyi, Big Data and Audit Evidence, Journal of Emerging Technologies in Accounting 12 (2015) 1–16. URL: https://doi.org/10.2308/jeta-10468. doi:10.2308/jeta- 10468. [13] A. Eilifsen, F. Kinserdal, W. F. Messier, T. E. McKee, An Exploratory Study into the Use of Audit Data Analytics on Audit Engagements, Accounting Horizons 34 (2020) 75–103. URL: https://publications.aaahq.org/accounting-horizons/article/34/4/75/2430/ An-Exploratory-Study-into-the-Use-of-Audit-Data. doi:10.2308/HORIZONS- 19- 121. [14] F. Krieger, P. Drews, P. Velte, Explaining the (non-) adoption of advanced data analytics in auditing: A process theory, International Journal of Accounting Information Systems 41 (2021) 100511. URL: https://www.sciencedirect.com/science/article/pii/S1467089521000130. doi:10.1016/j.accinf. 2021.100511. [15] A. A. Austin, T. D. Carpenter, M. H. Christ, C. S. Nielson, The Data Analytics Journey: Interactions Among Auditors, Managers, Regulation, and Technology, Contemporary Accounting Research 38 (2021) 1888–1924. URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/1911-3846.12680. doi:10.1111/1911- 3846.12680, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/19113846.12680. [16] V. Arnold, S. Sutton, The theory of technology dominance: Understanding the impact of intelligent decision aids on decision makers’ judgments, Advances in Accounting Behavioral Research 1 (1998) 175–194. [17] S. G. Sutton, V. Arnold, M. Holt, How Much Automation Is Too Much? Keeping the Human Relevant in Knowledge Work, Journal of Emerging Technologies in Accounting 15 (2018) 15–25. URL: https://publications.aaahq.org/jeta/article/15/2/15/9279/ How-Much-Automation-Is-Too-Much-Keeping-the-Human. doi:10.2308/jeta-52311. [18] V. Arnold, P. A. Collier, S. A. Leech, S. G. Sutton, Impact of intelligent decision aids on expert and novice decision-makers’ judgments, Accounting & Finance 44 (2004) 1–26. URL: https: //onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-629x.2004.00099.x. doi:10.1111/j.1467-629x. 2004.00099.x, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-629x.2004.00099.x. [19] J. L. Bierstaker, D. Hanes-Downey, J. M. Rose, J. C. Thibodeau, Efects of Stories and Checklist Decision Aids on Knowledge Structure Development and Auditor Judgment, Journal of Information Systems 32 (2018) 1–24. URL: https://doi.org/10.2308/isys-51913. doi:10.2308/isys-51913. [20] C. Dowling, S. A. Leech, R. Moroney, Audit Support System Design and the Declarative Knowledge of Long‐Term Users, Journal of Emerging Technologies in Accounting 5 (2008) 99–108. URL: https://doi.org/10.2308/jeta.2008.5.1.99. doi:10.2308/jeta.2008.5.1.99. [21] J. M. Rose, B. A. McKay, C. S. Norman, A. M. Rose, Designing Decision Aids to Promote the Development of Expertise, Journal of Information Systems 26 (2012) 7–34. URL: https://doi.org/10. 2308/isys-10188. doi:10.2308/isys-10188. [22] V. Arnold, P. A. Collier, S. A. Leech, S. G. Sutton, A. Vincent, Incase: Simulating Experience to Accelerate Expertise Development by Knowledge Workers, Intelligent Systems in Accounting, Finance and Management 20 (2013) 1–21. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/isaf. 1337. doi:10.1002/isaf.1337, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/isaf.1337. [23] R. Parasuraman, T. Sheridan, C. Wickens, A model for types and levels of human interaction with automation, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 30 (2000) 286–297. URL: https://ieeexplore.ieee.org/document/844354. doi:10.1109/3468.844354, conference Name: IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. [24] R. Parasuraman, V. Riley, Humans and Automation: Use, Misuse, Disuse, Abuse, Human Factors 39 (1997) 230–253. URL: https://doi.org/10.1518/001872097778543886. doi:10.1518/ 001872097778543886, publisher: SAGE Publications Inc. [25] R. Parasuraman, C. D. Wickens, Humans: Still Vital After All These Years of Automation, Human Factors 50 (2008) 511–520. URL: https://doi.org/10.1518/001872008X312198. doi:10.1518/ 001872008X312198, publisher: SAGE Publications Inc. [26] R. Parasuraman, D. H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52 (2010) 381–410. URL: https://doi.org/10.1177/ 0018720810376055. doi:10.1177/0018720810376055, publisher: SAGE Publications Inc. [27] K. M. Volz, M. C. Dorneich, Evaluation of Cognitive Skill Degradation in Flight Planning, Journal of Cognitive Engineering and Decision Making 14 (2020) 263–287. URL: https://doi.org/10.1177/ 1555343420962897. doi:10.1177/1555343420962897, publisher: SAGE Publications. [28] J. Bédard, Expertise in auditing: Myth or reality?, Accounting, Organizations and Society 14 (1989) 113–131. URL: https://www.sciencedirect.com/science/article/pii/0361368289900378. doi:10.1016/ 0361-3682(89)90037-8. [29] S. G. Sutton, V. Arnold, M. Holt, An Extension of the Theory of Technology Dominance: Capturing the Underlying Causal Complexity, 2022. URL: https://papers.ssrn.com/abstract=4101835. doi:10. 2139/ssrn.4101835. [30] L. G. MILITELLO, R. J. B. HUTTON, Applied cognitive task analysis (ACTA): a practitioner’s toolkit for understanding cognitive task demands, Ergonomics 41 (1998) 1618–1641. URL: https: //doi.org/10.1080/001401398186108. doi:10.1080/001401398186108, publisher: Taylor & Francis _eprint: https://doi.org/10.1080/001401398186108. [31] H. Vu, J. Haase, H. Leopold, J. Mendling, Towards a Theory on Process Automation Efects, in: C. Di Francescomarino, A. Burattin, C. Janiesch, S. Sadiq (Eds.), Business Process Management Forum, Springer Nature Switzerland, Cham, 2023, pp. 285–301. doi:10.1007/978-3-031-41623-1_ 17. [32] T. Rinta-Kahila, The University of Queensland Business School, E. Penttinen, Aalto University School of Business, A. Salovaara, Aalto University / University of Helsinki, W. Soliman, University of Agder, Norway / University of Jyväskylä, J. Ruissalo, Aalto University School of Business, The Vicious Circles of Skill Erosion: A Case Study of Cognitive Automation, Journal of the Association for Information Systems 24 (2023) 1378–1412. URL: https://aisel.aisnet.org/jais/vol24/iss5/2/. doi:10. 17705/1jais.00829.