The enthusiasm surrounding the use of artificial intelligence (AI) enabled digital solutions in healthcare is tempered by uncertainty around how it will change the working lives and practices of clinicians and healthcare professionals. To date, research regarding burnout has mainly focused on clinicians, with limited emphasis on other healthcare staff and a holistic understanding of AI-enabled automation adoption. The aim of this paper is to outline the method used to develop, refine, and validate a survey instrument that is able to evaluate the impact of automation on the healthcare workforce.
method
Within healthcare, artificial intelligence (AI) enabled automation solutions have recently become more prevalent, promising to increase productivity and reduce clinical workloads [1]. These initiatives have already demonstrated their ability to perform activities such as reading scans or completely replacing clinical consultations that were previously solely within the domain of human clinicians [24]. Research suggests that although new technologies have the ability to undertake clinical activities, their full potential is not always realised, and the desired outcomes are not observed, deeming such initiatives as unsuccessful. Studies have so far revealed several factors that can contribute to low technology acceptance and adoption, which negatively impacts workloads and, consequently, clinician wellbeing2. It is therefore critical that a comprehensive understanding of the interactions between people, technologies, and the system is taken into account for successful adoption and implementation , while ensuring staff retention and reducing turnover [5]. The World Health Organisation (WHO) projects a global needs-based shortage of health care workers at over 14.5 million by 2030 [6]. In light of this, a holistic understanding is crucial because employees whose roles are directly affected by automation or digitisation can either be empowered to 'work at the top of their licence' or feel even more disempowered and left behind.
In the UK National Health Service (NHS), the workforce crisis has been deemed the “biggest, most pressing threat to the viability of services for people who need them” [7]. Studies have observed a bidirectional link between burnout and medical errors resulting in clinician distress, whilst conversely, better physician wellbeing was associated with improved patient satisfaction, improved treatment, and
2023 Copyright for this paper by its authors. CEUR Workshop Proceedings
ceur-ws.org ISSN1613-0073 lower rates of hospital-acquired infections [8]. It is argued that improving healthcare workforce wellbeing will benefit not only the individual clinician but impact patient care and safety, whilst reducing provider costs [9].
Several studies on digital health technologies focus on patient outcomes and satisfaction, yet a dearth of literature exists on the wider impacts such as those on different members of the healthcare workforce (i.e., managers, clinicians, nurses, administrators). Recently, some studies have investigated healthcare professionals’ perceptions of AI [e.g., 10–12], yet a questionnaire that measures perceptions and perceived impact of AI enabled automation does not yet exist. Therefore, we used the modified e-Delphi approach3 to develop and validate a questionnaire that explores the impact of AI-enabled automation on healthcare professionals and their perceptions both pre- and post-implementation. The questionnaire aims to investigate domains such as trustworthiness and acceptability of the technology, as well as capturing data around staff wellbeing, enabling a holistic understanding of perceptions and the impact of AI-enabled automation adoption. 1.1.
In most settings, newly introduced change is observed with mixed attitudes and perceptions by users to whom the change impacts. While change acceptance is not regarded as an easy process and is often met with resistance, in order to improve workflow and efficiency in the long term, acceptance is a fundamental element for technology adoption and implementation and, hence, understanding the differing perceptions could facilitate acceptance and use.
In the Information Systems (IS) and health informatics community, most studies of user acceptance and adoption of innovative technological initiatives are based on the Technology Acceptance Model (TAM), Theory of Planned Behaviour (TPB), Diffusion of Innovation Theory (DOI), and Unified Theory of Acceptance and Use of Technology (UTAUT) [13]. Each framework offers a slightly different perspective to acceptance. For example, the TAM aims at predicting behaviour attitudes towards a specific technology (perceived usefulness, perceived ease of use), TPB provides a predictive and explanatory model of attitudinal-behaviour reactions based on three factors (personal attitude, perceived social norm, perceived behavioural control), and UTAUT considers individual perspectives and the influence of environmental and social factors on technology including performance expectancy, effort expectancy, social influence, and facilitating conditions, such as legal liability, organisational culture, and organisational infrastructure.
Despite this, there is general consensus that attitudes towards AI, differs from traditional technology acceptance frameworks. The General Attitudes towards Artificial Intelligence Scale (GAAIS) [14] and Technology Readiness Index (TRI) [15] measure individuals’ characteristics, such as ‘insecurity’, ‘discomfort’, ‘innovativeness’, and ‘optimism’, that may influence technology acceptance. Nevertheless, these do not examine perceptions that healthcare staff have prior to implementing AI technologies (i.e. pre-implementation). Additionally, they do not measure a healthcare worker's sense of readiness or their perceptions of how AI-enabled automation will affect their professional role/ work practices.
Whilst already validated technology acceptance models could yield valuable insights, the socio technical system (STS) approach adds another dimension by acknowledging the interdependencies existing between organisations, individuals, and technology. The STS approach implies that the technical and social subsystems of work cannot be decoupled and are inter-related; the compatibility and interaction between the two subsystems determine the effectiveness of a specific work system. While the technical subsystem is concerned with “the processes, tasks, and technology needed to transform inputs to outputs,” the social subsystem is concerned with “the attributes of people (e.g. attitude, skills, values), the relationships among people, reward systems, and authority structures” [16, p.17]. The interactions between these two subsystems produce the outputs of a work system, creating economic outcomes such as cost reduction, efficiency, and productivity effectiveness, as well as humanistic outcomes such as wellbeing and engagement. 3 We adopted a modified e-Delphi approach by starting the process with a set of selected items drawn from numerous sources, including the existing literature, and theoretical framework.
Socio-technological studies on healthcare and technology implementation found that stakeholders
hold different perspectives depending on their goal and expectations. For example, the organisation's
goal of addressing workload productivity or maximising financial performance may be different from
a healthcare professional's focus on improving patient outcomes and clinical decision -making. When
introducing new technology into the healthcare industry, the challenge is in determining the best method
for comprehending the perceptions of all stakeholders [
This study adopts the STS perspective as a theoretical framework to analyse the impact of AI enabled automation on the healthcare workforce and their work practices. 1.2.
The Delphi approach has become an increasingly popular tool used in both IS research [18] and healthcare [19], to evaluate current knowledge, formulate methodological or theoretical guidelines, formulate recommendations for action and prioritising measures, resolving controversy in management, and develop assessment indicators and tools. It is argued that this approach is useful in problematic areas where there is a lack of consensus among experts, or expert judgement is preferable to individual opinion, which is in line with the rapidly evolving nature of the healthcare context and digita l transformation [20].
The Delphi methodology is a multi-step process where each stage builds on the outcomes of the prior stage. It involves giving participants rounds of questionnaires, whereby the responses to each questionnaire are analysed and evaluated before creating a refined one used for the following stage. The process (i.e., rounds) continues until (a) consensus is obtained, or (b) opinions are clarified. There are often concerns with respect to the number of rounds that is required for consensus to be obtained. Some scholars have noted that the classical or traditional method often employs three or more rounds. However, the general assumption now appears to be that two or three rounds are preferable given that participants may become fatigued if the process is longer, and usually stability and consensus should have been attained after three rounds [21]. For this study, a group agreement of 75% or greater on each question was an acceptable level of consensus, based on Diamond et al.’s [22] systematic review.
The usage and modification of the Delphi method have led to the emergence of numerous forms of Delphi research, including, for example, “classical Delphi”, "modified Delphi," "e-Delphi," "Delphi policy," and "Real-time Delphi". It is argued that no matter which ‘type’ is favoured, the generic aim of the approach is to determine, predict and explore group attitudes, needs and priorities (although not always striving to achieve consensus) regarding a specific problem area [23].
A scoping review of the literature was conducted to identify any previous questionnaires developed to explore burnout and the impact of AI or automation on clinicians and healthcare professionals, and if not, then to identify possible key constructs for the survey. The search was conducted independently in 2022 by two of the authors using the following databases Google Scholar, PubMed, Web of Science, and Scopus, to cover content in both healthcare and general sources. A combination of various keywords was used, including burnout, wellbeing, AI, clinicians, and healthcare. We used the snowball sampling method by manually reviewing the papers’ reference lists we identified that might consist of further relevant references. We included peer-reviewed empirical studies as well as theoretical or systematic literature review studies (written in English) focusing on the perception and impact of AI (or technology) and burnout/ wellbeing on individuals and work practices.
While we identified several validated burnout instruments used previously in the healthcare setting (e.g., Maslach Burnout Inventory (MBI), Stanford Professional Fulfilment model, etc.), to date, there is no validated questionnaire that explores the perceptions and impact of AI-enabled automation on different members of the healthcare workforce. 2.1.
In this research, and in line with the e-Delphi design type, we used online questionnaires, and faceto-face or virtual meetings to facilitate clarification and discussion.
Participants who showed interest in or involvement in AI in the disciplines of IS and healthcare were chosen to be part of the study panel. Healthcare professionals (both those with and without an interest in health technology) were candidates for the panel, as well as scholars with more than 10 years of experience in IS and technology adoption in the healthcare sector. It is argued that having a varied panel can offer an unbiased assessment of perception and current understanding in the particular fields o f investigation.
Panel members were recruited via email to take part in the e-Delphi process. To eliminate the inherent bias (e.g. dominance) and groupthink (i.e., group conformity) observed with face-to-face group meetings, the participants were known to the researchers but remained anonymous to other panel members, especially in the first round [20]. 2.3.
Each participant was sent an email outlining the research background, research questions, and a copy of the questionnaire. Four to six weeks later, participants were emailed again with the ‘round two’ survey, and then four to six weeks later, participants were emailed again with the ‘round three’ survey.
Round 1 concentrated on identifying issues and commenting on the structure and content of the survey developed. Preliminary questions of the survey were developed using the theoretical framework, a validated burnout survey, and previous questions used to investigate the impact of AI on the individual level and work practices. The panel had the freedom to suggest alternative questions, delete initial questions, or provide comments of thoughts and ideas. Panel members had around 2-3 weeks to respond and returned their comments to the researchers.
At the end of Round 1, ideas and suggestions were consolidated and a meeting was held to clarify any suggestions/ feedback and to facilitate discussion. The questionnaire was then amended in line with this and sent back to panel members for Round 2 of the process. Since there was no previous study quite similar to this (i.e., investigating the impact of AI-enabled automation on different members of staff depending on their involvement with the intervention), there was some debate among participants regarding constructs, wordings, and relevance in Round 2.
Round 3 involved distributing the final survey draft for panel review for any additional comments or revisions.
Between December 2022 and March 2023, eight healthcare staff (both clinical and clerical expertise) and five scholars participated in three rounds of the study to reach a consensus on the terminology, structure, and content of the survey that was designed to explore healthcare professionals' perceptions of AI-enabled automation in routine clinical conversations.
Elements were added or removed, and others were adjusted for clarity or increased in depth based on the panel recommendations during Round 1 of the Delphi review. Experts showed a higher level of agreement on the bulk of the items after Round 2 of the process. This shows the significance of the Delphi process in forming broadly recognised consensus by taking into account the comments and recommendations of our interdisciplinary panel members. Round 3, which consisted of the dichotomous "keep" or "discard" for each item, offers confidence that the items were not changed or had their broad applicability restricted by the Round 2 changes.
Table 2 provides a summary of the final survey components and description.
Description Information about the hospital, job role, experience, work hours, age, gender, and ethnicity.
Assesses the participant's role in the pathway and their self-rated knowledge of the system. Measures intrinsic positive reward derived from work and symptoms of work exhaustion and disengagement.
Evaluates the perceived impact of the system on burnout symptoms.
Gauges the acceptability of the system, including user satisfaction and the system's perceived value.
Explores perceived benefits, performance anxiety, communication barriers, benefits, privacy concerns, liability, and risks.
The aim of this study was to develop and validate a survey instrument that captures the perceptions of healthcare professionals with regards to AI-enabled automation, specifically used in this context.
In order to reach a consensus regarding structure and content, the Delphi technique involves repeatedly surveying participants for their thoughts on a certain subject. Experts in the subject of inquiry make up the participants (or panellists) in a Delphi research. Therefore, healthcare professionals as well as specialists in the field of health informatics including academics and researchers active in IS practice or research, made up the participants for this e-Delphi approach. The e-Delphi approach was used to identify whether healthcare and academic experts could reach consensus on constructs used to explore perceptions and impact on clinicians and healthcare professionals. After three rounds, the results were stable. Although the recommended number of Delphi rounds varies across the literature, 2-3 rounds are typical and argued to be sufficient. We chose to perform three rounds of the study in order to maintain response consistency. More rounds would have been taken into consideration, though, if consensus was not obtained after three rounds.
This study has a few limitations. The Delphi technique requires significant time and effort in terms of, for example, creating evaluation checklists and amending the questionnaires after each Round. In some instances, the research team had to contact some panel members to request particular input since they had missed some items, which could have influenced their overall responses. Also, we note that the meetings with clinical and academic backgrounds were held separately due to participant availability. Furthermore, although a minimum of 75% indicates consensus among our panel members, greater consistency on some items would make the findings more compelling. In addition, we note the study's sample size of 13 participants, potentially limiting its ability to be generalised.
The authors would like to thank all participants for their time and constructive feedback.
This work is supported by The MPS Foundation Grant Programme. The MPS Foundation was established to undertake research, analysis, education and training to enable healthcare professionals to provide better care for their patients and improve their own wellbeing. To achieve this, it supports and funds research across the world that will make a difference and can be applied in the workplace.
SK, AH, EL, NdeP are employees of Ufonia Ltd.
Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018;2(10):719–731.
Benjamens S, Dhunnoo P, Meskó B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ Digital Medicine; 2020;3(1):1–8. Khavandi S, Lim E, Higham A, de Pennington N, Bindra M, Maling S, Adams M, Mole G. User-acceptability of an automated telephone call for post-operative follow-up after uncomplicated cataract surgery. Eye; 2022;1–8.
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature; 2017;542(7639):115–118 Mckinley N, Mccain S, Convie L, Clarke M, Dempster M, Campbell WJ, Kirk SJ. Resilience, burnout and coping mechanisms in UK doctors: a cross-sectional study. BMJ open, 10(1) World Health Organization (WHO). Health workforce requirements for universal health coverage and the sustainable development goals. (Human Resources for Health Observer Series