Pandemics or health emergencies create situations where the demand for clinical resources greatly exceeds the supply leading to health providers making morally complex resource allocation decisions. To help with these types of decisions, health care providers are increasingly deploying artificial intelligence (AI)-enabled intelligent decision support systems. This paper presents a synopsis of the current debate on these AI-enabled tools to suggest that the existing commentary is outcome-centric i.e. it presents competing narratives where AI is described as a cause for problematic or solution-oriented abstract and material outcomes. Human decision-making processes such as empathy, intuition, and structural and agentic knowledge that go into making moral decisions in clinical settings are largely ignored in this discussion. It is argued here that this process-outcome divide in our understanding of moral decision-making can prevent us from taking the long view on consequences such as conflicted intuition, moral outsourcing and deskilling, and provider-patient relationships that can emerge from the long-term deployment of technology devoid of human processes. To preempt some of these effects and create improved systems, researchers, providers, designers, and policymakers should bridge the process-outcome divide by moving toward human-centered resource allocation AI systems. Recommendations on bringing the human-centered perspective to the development of AI systems are discussed in this paper.
Resource allocation is a type of decision-making which
involves the procurement, assignment, and distribution of
resources between actors. Typically, decision-making
pertaining to resource allocation is considered to be a difficult
enterprise because access or ownership of resources is
competitive and can not only affect individuals and group health,
but a set of interconnected socio-economic variables.
Furthermore, this type of decision-making has a moral
dimension as it requires humans (e.g. physicians, hospital
administrators, nurses, etc.) to make tradeoffs such as those
involving utilitarian and egalitarian parameters which
exacerbates its comp
One of the ways providers are making resource allocation
decisions to deal with the COVID-19 pandemic is to use
AIenabled decision support systems. These systems can use
patient’s electronic health records (EHR) and/or clinical
measurements (e.g. blood pressure, fever, health conditions)
to make diagnoses and prognoses (
Since the technology can affect people’s lives in multiple ways, it renders itself as a problem concerning social good, and thus deserves our attention. Furthermore, as it becomes more sophisticated and widely deployed, it needs to be understood now to preempt any negative consequences on human decision-making practices and to generate helpful policies and guidelines. To enhance our understanding on this matter, this paper presents a brief synopsis of the debates surrounding the use of AI to help humans make moral decisions pertaining to resource allocation during health crises. The core argument made here is that the current debate on the deployment of AI-enabled decision support systems comprises competing narratives that are mostly outcomecentric i.e. they focus on the abstract (e.g. fairness) and material (e.g. saving costs) outcomes AI technology can yield. By emphasizing upon these outcomes, we fundamentally ignore the human decision-making processes that health care providers also use in addition to established clinical guidelines when allocating resources. The disregard of human processes of decision-making in the pursuit of AI’s use in resource allocation might have long-term consequences on the design of technology and providers’ abilities to make decisions. My hope is that this paper will provide insights to researchers, designers, and policymakers to bridge processoutcome divide to implement more human-centered technology for moral decision-making in clinical settings.
To support these arguments, the following sections begin by presenting a brief synthesis of current perspectives on using AI for resource allocation during COVID-19. We observe that the predominant ideas on making resource allocation decisions reflect a process-outcome divide where the current debate is heavily tilted toward framing AI as an entity that can create outcomes which are either problematic or solution-oriented in nature. This approach disregards human processes such as empathy, intuition, and structural and agentic knowledge that health care providers rely upon to make resource allocation decisions. This is followed by noting the possible effects of long-term deployment of outcome-centric technology on human decision-making such as disruptions in intuitive processes, moral outsourcing and deskilling, and relationships with patients. It is recommended here that the process-outcome divide must be bridged by creating human-centered AI systems which can preempt adverse consequences for providers and patients. Human-centered AI systems in health care can be incorporated by having providers work as co-developers of technology, building in-house AI capabilities, and developing regulations pertaining to the use of AI in health care settings.
Fairness is a recurrent theme in the discussion on the use of
AI to make moral decisions. The concept of fairness when
applied to any resource allocation process refers to the
instance whereby a decision results in reduction of biases that
may prioritize one group over another or where it increases
equity between different stakeholders. The goal of any
decision-maker, therefore, is to increase fairness as an outcome.
However, the debate on the use of AI with reference to
fairness is a debate comprising competing narratives. Some
argue that AI-based tools in health care resource allocation
during COVID-19 pandemic are useful because machines
are driven by complex logic and predetermined parameters,
and therefore, can be fair and less biased resource a
However, a competing narrative suggests the opposite. It
argues that AI systems can create problems as they may
reflect existing systemic biases and thus, are more likely to
make unfair appraisals exacerbating inequality between
different racial and socioeconomic groups
In addition to fairness, another theme concerning the use
of AI tools in health care resource allocation pertains to the
effects of AI’s computational prowess on various abstract
and material outcomes. It has been argued that unlike
humans, machines have extraordinary computing and
information processing power which allows them to gather,
analyze, and interpret data quickly which ultimately helps with
timely diagnosis and provision of care (Shea et a
It is evident that current perspectives discussed above on deploying AI to make resource allocation decisions are concerned with tackling abstract (e.g. fairness) or material outputs (e.g. costs). Such framing is not only outcome-centric, but overly simplifies complex phenomena concerning moral decision-making such as resource allocation. The application of AI-enabled technology to allocate resources occurs in human contexts where decision-makers use distinct and identifiable processes to make moral decisions. An AI-enabled system neither accounts nor substitutes for these processes and therefore, the need for identifying and discussing human processes becomes ever more important to not only fill theoretical voids, but also affect how we design and implement technology.
Clinical decision-making in the realm of resource allocation is a multifaceted activity which requires providers to use complex decision-making processes in addition to predefined scientific and well-established protocols. The sections below present a brief discussion on the distinct processes which are regularly employed in moral decision-making by health care providers but are yet to be explored within the context of resource allocation especially during a pandemic.
Intuition refers to an information processing mode which
lacks conscious reasoning but incorporates affective and
cognitive elements to make decisions (Sinc
Health providers often have a deep understanding of structural and agentic variables that underlie and affect their dayto-day operations. For instance, they may know how their hospital’s location affects resource procurement, patient arrival and admission. Providers are also more are more likely to be aware of differences in hospital personnel personality types, work ethics, interpersonal relationships, cultural values and political beliefs, bureaucratic procedures and administrative conduct, equipment issues, etc. Together, this conscious and subconscious knowledge of structural and agentic factors can guide providers’ moral decision-making and thus, how they allocate resources (Lemieux-Charles et al. 1993). The current technology can hardly substitute this knowledge as it exists beyond the purview of an AI-enabled tool.
Many diagnostic and moral decisions are driven by empathetic concern for others (Selph et al. 2008). Hence, it is no surprise that care providers often take an empathetic approach to identify illnesses or allocate specific resources. Empathetic concern plays are a very important role in identifying biases in any system, policy, and practice. For instance, when we reflect on a process empathetically, we are more likely to understand how it affects people which can therefore allow us to intervene to help and make changes for them (Batson 2016). To illustrate this further, let’s reconsider the findings on the use of algorithm which led to Black patients being given access to fewer resources despite them being sicker since the tool factored the costs of health care more heavily when allocating services (Obermeyer et al. 2019). If the AI program were developed using an empathetic approach, then it might have accounted for the fact that Blacks on average have lower income than White patients and thus, are less likely to spend on hospital services despite having more physiological symptoms. This shows that the design and use of technology to make moral decisions without any concern for empathetic human processes can reflect in the outcomes technology produces.
While intuition, structural and agentic knowledge, and empathetic concern are important processes that help and guide moral decision-making in clinical (and non-clinical settings), they are largely ignored in the debate on the use of AI to allocate resources since competing narratives are mostly focused on the outcomes the technology produces. This raises the question of what the deployment of AI means for human processes in decision-making. To help researchers, administrators, and policy makers engaged in long-term planning and thinking, I present some reflections on the possible effects of AI-enabled tools devoid of elements concerning human processes.
The focus of this paper is on moral decisions pertaining to resource allocation especially within pandemic-related settings. Since many people compete for limited resources and there is little time to make decisions; it is tempting, and in some cases, advantageous to apply AI-based tools. However, when humans use AI-enabled tools to make moral decisions, their internal decision-making processes are likely to be affected or influenced by technology. This could affect how providers assess, analyze, and treat patients. However, short-term solutions can potentially have long-term unintended and unwanted effects. The following sections note some of the consequences on providers’ decision-making processes concerning intuition, knowledge, and empathetic concern which may occur as a function of long-term deployment of AI.
As AI continues to be incorporated in moral
decision-making, providers will have to divide their attention between the
AI’s recommendations and their own intuitive judgement
especially if they have different or opposing ideas. They will
face the added tension of determining tradeoffs between the
machine and their own mora
It is arguable that the addition of AI may could facilitate the doctors’ decision-making processes by sharing the cognitive burden pertaining to diagnostic evaluation. However, it is important to note that the process of moral decisionmaking comprises more than a more than a mechanical diagnostic endeavor. It also includes how users react to and accept or react suggestions from AI. Prior research has shown people’s tendency to both accept and reject advice from algorithms (Dietvorst et al. 2014; Logg et al. 2019) and therefore, it is likely that such judgements pertaining to the recommendations made by AI will also be made by doctors in conjunction with their own intuitive responses.
Disrupted and conflicted intuition is likely to affect the internal moral compass decision-makers use to organize their worlds. It is also going to reflect in how they allocate resources where some may exclusively rely on technology to mitigate their internal tensions and others may develop their own course of action. Although it is possible that providers use a combination where they select when to choose intuitive or machine judgement to make decisions, this will be a hard skill to learn, and thus, difficult to use especially in emergencies where decisions have to be made quickly.
Assigning and rationing resources between people is an
issue that is directly tied to the issues of ethics and morality.
Making moral decisions can be a difficult and distressing
process because it typically involves trade-offs concerning
self-interests and group needs, personal and cultural values,
and immediate and future rewards within the context of
health care
A patient’s relationship with their health provider is based
on several factors including the providers’ abilities to
empathize and make decisions that show off their competence,
expertise, and clinica
ward the Development of Human-Centered AI
Now that we have identified that there is a process-outcome divide on how moral decision-making is conceptualized, discussed, and applied within clinical settings, the next question is what we can do about it? The bridging of the process-outcome divide in moral decision-making can occur with the development of human-centered resource allocation AI systems as applied to clinical settings. A human-centered approach to AI development incorporates the perspectives and processes of users that use intelligent systems (see Xu 2019). Thus, AI systems using a human-centered approach are more likely to create synergy between both human decision-making processes and machine outcomes to positively affect and amplify both physician and patient welfare. That being said, the challenge remains as to how we can achieve human-centered design in the deployment and development of AI tools.
To overcome this challenge, we first need to further unpack the process-outcome divide in the context of moral decision-making pertaining to resource allocation within a pandemic (or non-pandemic) setting. The “process” in the process-outcome divide refers to the human decision-making processes such as empathy, intuition, and structural and agentic knowledge which health care providers use (in addition to pre-determined clinical protocols and guidelines) to make diagnostic judgements and allocation decisions. While the “outcome” refers to the machine-driven or related consequences or functionality such as maximizing fairness or computational capacity. Hence, process-outcome divide by nature can be said to also imply a human-machine divide. Note that here the term human-machine divide is not meant in the same way as its prior use in the context of technical features of the machine and how they are informed by human biology (e.g. neurons) (see Warwick 2015). The focus here is on how the process-outcome divide in perspectives on moral decision-making pertaining to resource allocation reflect the split between human processes of decision-making and machine-generated outcomes pertaining to resource allocation decisions. I argue that this divide could potentially be addressed by moving toward human-centered AI systems which will require recognizing and iteratively integrating human processes in the development, deployment, management, and regulation of AI-enabled systems. To this end, the following sections present some recommendations on how human processes can be weaved into the development of AI systems.
An admittedly simplified way of understanding how
AI-enabled technology is scaled is to reflect on two stages:
development and deployment. More often than not, these tools are
developed either independently (i.e. by
manufacturer/company or within academic settings) or in some consultation
with health care providers. Once developed, they may be
pitched to various health care providers where the
technology is customized to their needs. Sometimes, the technology
is rolled out in phases where it is tested on a smaller level
and subsequently expanded to include more patients or units
(Gago et a
This approach to the scaling of an AI-enabled decisions support system may seem natural and functional. However, I argue that for including human processes of decision-making in how technology is used; it is best to see development and deployment linked together in an interactive and iterative process where they inform each other. This is particularly important in health care settings where the availability of and access to resources vary and the environment (e.g. infection and mortality rates, deaths, policy, information) change rapidly and often unpredictably.
To creative an iterative loop between development and deployment, the lines between and users of technology have to be blurred. While developers of technology can understand its various technical aspects and have the requisite knowledge and skills to build it; the users can often envisage its effects and uses more deeply due to their day-to-day experience, exposure to patients’ needs, and structural and personnel-based issues in clinical settings. To understand this further, let us imagine that an AI program is designed to help decide if patient gets a bed during a pandemic. The program conducts a risk assessment of the severity of patient’s condition by assigning scores on a pre-determined set of factors. One of the factors relates to prior health condition where the AI assigns a score in case a patient has any (e.g. heart problem). However, health care providers may know via their day-to-day experiences that it is likely for a patient without a prior health record in the hospital’s system and yet having an underlying condition to arrive in the emergency department. The patient might be unaccompanied and unable to report their medical history due to physical ailment or language barrier. They could also be unaware of their underlying medical condition. In such a scenario, the use of an AI program that determines if a resource (e.g. bed.) can be allocated to a patient based on the above-mentioned criterion may not be the most appropriate option. If providers and developers work in an interactive and iterative fashion, then these observations could be passed on to the developers who may be able to account for these issues i.e. a lack of prior medical record, or language barrier, or being unaccompanied along with obvious severity in symptoms when assessing risk. An AI program could then use a different scoring system which accounts for these variables to allocate beds. Thus, continual integration of human processes via relevant updates and modifications is more advantageous than one-time testing or multi-phase testing with a pre-determined end.
Prior research has shown that users and developers can co-create technology by contributing their differing expertise in a process called cooperative prototyping (Bødker and Grønbaek 1991). However, the rapidly changing health environments require us to leap from the cooperative prototyping approach to iterative cooperative development and management of technology. When health care providers will act as co-developers of technology, it will allow them to fuse human processes (e.g. empathy, intuition) involved in moral decision-making to build and update AI systems.
That being said, it must be mentioned that from normative and prescriptive perspectives, the extension of providers as co-developers of technology may sound like an appealing and useful idea. However, from a practical point of view, this may prove to be a difficult enterprise because it would require health care facilities to dedicate personnel and their time toward the development of such systems. Presumably, this thought may be a deterrent for some to adopt such measures and protocols. However, I argue that in the long run, this will be a small cost to bear. A team of health professionals who are dedicated to testing AI-enabled intelligent decision support systems can only better the technology which in turn will produce superior outcomes and decrease the cost of day-to-day operations as well as reduce risks associated with poorly designed systems. Consider this with reference to the latest developments in space science. IBM developed a robot called CIMON which was tested for its efficacy by an astronaut aboard the International Space Station (ISS) (CIMON brings AI to the International Space Station n.d.). The feedback and testing allowed for a new and upgraded robot CIMON-2 to be sent to the ISS (IBM 2020). The developers and user (i.e. astronaut) of the space robot played important roles in both the development and deployment of the technology before it could be used in a highstakes environment such as a space mission. Health care settings should be treated no less than a space mission as they are high-stakes and expense-laden environments which affect socioeconomic and mortal outcomes for billions of people around the world. It logically follows then AI-enabled decision support systems within health care should not only be tested regularly but be informed by the very users who employ it making critical resource allocation decisions.
As decision-makers, people are managed by others such as human resource departments, administrative procedures or protocols, upper managements, etc. These institutional actors and protocols manage human activities, solve issues, and recommend further actions. Management also extends to medical equipment as hospitals and clinics often have technical staff or support teams on site or procured via thirdparty contracts. However, such staff or administrative units are often missing when it comes to overseeing AI-enabled technology. Many health care settings deploy AI with little to no oversight of these systems since their management requires particular skill sets. Increasing sophistication of AIenabled systems and their authority to pass judgement (assign risks to patients/calculate scores) makes them not only tools but also decision-making actors to some extent. As actors and tools, they too, need supervision. Therefore, health care administrators will need to develop in-house expertise and create departments that are specifically dedicated to the monitoring, modification, and management of AI-enabled systems or embodied intelligent assistants such as robots which have increasingly become a part of health care settings.
Such an endeavor would have several benefits as it would: a. allow the integration of providers’ perspectives within the AI system, b. identify any issues quickly, and c. make modifications to the system potentially within the clinical settings or outsource them to the third-party or original developers within a short period of time.
It was argued above that one of the long-term consequences of using AI to make moral decisions could manifest as humans putting in more cognitive effort and challenging their intuition especially if personal judgement were at odds with the advice given by an AI program. It was also suggested that an over-reliance on intelligent systems could lead to moral outsourcing and deskilling. To preempt such scenarios, health care providers will need to create specific protocols, regulatory, and ethical guidelines to regulate the use of AI within their premises. These guidelines and directives will need to specify whose judgment—i.e. human or AI— will be considered as the final say when making a diagnostic or allocation decision. These guidelines will also have to delineate parameters of assigning culpability and responsibility if choices made by a human in conjunction with or against the advice of AI results in adverse outcomes. These regulations will help providers understand their roles in moral decision-making and allow them to continue sharpening their skills when it comes to moral decision-making in the presence of AI.
Additionally, medical schools and educational programs will also need to train providers and students on how to interact with AI, evaluate its judgement, and effects on human decision-making. Together, these practices will allow providers to better understand AI, its management, and relationship with humans within clinical settings.
The core argument presented in this paper is that the discussion on AI decision support tools used in moral decisionmaking such as resource allocation within clinical settings provides competing narratives which delineate the pros and cons of AI in terms of the material and abstract outcomes the technology produces. Such perspectives distract us from focusing on the role human decision-making processes such as empathy, intuition, and structural and agentic knowledge play in resource allocation decisions. This scenario reflects process-outcome divide in the current perspectives on moral decision-making within health care settings. If these human processes are disregarded while AI is used to make moral decisions, it may result in long-term consequences such as conflicted intuition, moral outsourcing and deskilling, and poor patient-provider relationships. To preempt some of these consequences and create better health outcomes; researchers, developers and policymakers seriously consider the importance of human processes along with machinedriven outcomes. One of the ways we can bridge the process-outcome divide is to create human-centered AI systems specific to health care. To this end, some recommendations are proposed: a. health care providers work with developers of technology as co-developers in an in an iterative and interactive fashion, b. health care facilities should develop in-house AI expertise and create a specific department to manage, regulate, and modify the technology, and c. regulatory protocols and guidelines specific to the use of AI in making moral decisions should be developed. These guidelines should be able to specify how and when humans should override AI decisions. It should also delineate rules on culpability should a decision made in conjunction or against AI advice produce adverse effects. Furthermore, providers and students should be trained on understanding the effects of AI on their decision-making. Together, these endeavors could help with taking and implementing a broader and more human-centered perspective on the use of AI in health care to advance social good.