Artificial intelligence (AI) applications in healthcare offer the promise of improved decision making for clinicians, and better healthcare outcomes for patients. While technical AI advances in healthcare showcase impressive performances in lab settings, they seem to fail when moving to clinical practice. In this position paper, we reflect on our experiences of designing for AI acceptance and discuss three interrelated challenges to AI in clinical practice: technical feasibility, financial viability, and clinician acceptance. We discuss each challenge and their implications for future research in clinical AI. We encourage the research community to take on these lenses in collaboratively tackling the challenges of moving AI systems into real-world healthcare applications.
Over the last decade, there has been lots of excitement about
what AI might do for healthcare. AI offers the promise of
improved cancer diagnosis, faster discovery of new drugs,
and even personalization of patients’ healthcare experiences.
The transition to electronic health records has produced a
wealth of data ripe for mining. Interestingly, AI systems that
work great in computer labs largely fail when they move
to clinical practice, and the number one reason they fail is
the lack of adoption by clinicians
Our team has been investigating how AI might function more effectively in the enterprise: How AI systems might help professionals both make better decisions and also feel like they are becoming better at their jobs. We have done work in education, business, and healthcare. Currently, we are working on a project to identify how AI might improve ICU care. Can it automate mundane tasks? Can it discover low-value care? Can it detect deviations between standards of care and actual practice?
For this position paper, we reflect on what we have learned about AI acceptance in the workplace across many domains, and we tailor our insights to aspects most relevant to clinical practice. In our experience, AI only flourishes when it is technically feasible, financially viable, and acceptable or even desired by end-users. Research on clinical
AI has largely focused on some aspects of technical feasibility. Researchers have made many, stunning technical advances that confirm this is a great space for innovation. Little to no work has investigated how AI systems might pay for themselves within the complex landscape of healthcare reimbursement, and little work has explored when, where, and in what form AI inferences might be viewed as valuable by clinicians. Below we touch on each of these three areas that we feel must be addressed for AI to thrive in the clinic.
While there have been great technical advances around AI
in healthcare, much of the work is not clinically relevant
Technology follows a familiar adoption process. First, there is a need for technical capabilities. Once capabilities exist, there is a need to map these capabilities to situations that might benefit from this capability. Finally, there is a need for evidence that applying the new capability actually created a desirable benefit. Almost all AI innovation in healthcare is working on new capabilities. That is an important first step. But now, we need more work showing the capabilities actually map to authentic needs. When we have a body of work showing AI can address real needs, the next step will be deployment studies showing the new technology has real-world impact, that it improves health outcomes and lowers the cost of delivering care.
One driver of the disconnect between AI capability and
clinical need is the fact that most AI innovations in health
focus on reproducing the work of human decision-makers.
These are often approaching problems from a machine
learning perspective because the diagnosis (human
decision) and selected treatment (human decision) are well
documented. The labeled data exists for training a system. Most
healthcare decisions like these are “textbook”, they are
obvious to a clinical expert. And an AI system trained on this
data will work the best for textbook cases. This is not where
clinicians need help, unless the intent is to automate
clinicians out of existence. Clinicians need the most help with the
unusual cases, cases with high clinician uncertainty
In the ICU, patients on a ventilator receive input from the Interventionist, the ICU nurse, and the Respiratory Therapists (RT). The RTs will perform breathing tests on ventilated patients in the early morning. When the Interventionist arrives, they use the results of this test to decide if a patient should be extubated. But RTs and Interventionists do not always agree on who should get a breathing test, leading to situations where the doctor wants the results but the test has not been conducted. An AI system that tries to predict expert disagreement could raise this issue ahead of time, allowing the experts to make a decision before the window for decision making has closed. Situations like expert disagreement are only indirectly captured in EHRs, but they show real moments where clinicians would benefit from a machine prediction.
In our experience, AI researchers working in healthcare are most interested in working on difficult challenges. This helps researchers publish, as they can offer clear evidence that they have advanced the state of the art for AI and machine learning. However, by doing this, they most often overlook the low-hanging fruit, situations where a little bit of well-known AI might actually help accelerate or enhance clinical practice. In our ICU work, we noticed that clinicians must frequently input orders for new medication. This is not a difficult task, but it is a tedious task. As they type, they see a list of possible medications and doses they might be looking for, shown as an alphabetical list. Sometimes this helps. In examining this mundane, tedious task, we noticed that a list of medications ordered by frequency as opposed to alphabetically significantly reduced the task completion time by more than 50%. This is not a “sexy” innovation nor even a use of AI. But it does help to illustrate how the mundane labor of interaction with IT systems is largely ignored by data scientists and AI researchers.
In our work on AI innovation in the enterprise, we have observed that the biggest barrier for getting a new AI capability off of the whiteboard and onto a product roadmap is a strong business case. Software product managers want to know that the value of the innovation will be much higher than the development costs and the operational costs for the innovation. This can be challenging due to the near-monopoly held by the tiny number of EHR vendors and from the software development culture that has shifted to lean-agile, with a focus on making an MVP – the minimum viable product. Technical AI healthcare research never addresses how the advance will make money, how it will pay for itself. The work does not detail the changes needed to current data pipelines. It does not talk about the increased amount of computing required. It does not specify how this will save time, allowing clinicians to treat more people in the same amount of time, and it does not detail how clinicians might be able to charge more, because the quality of the decision-making should get better. The flow of money in healthcare is complicated, from patient to insurance company to the many intersecting clinicians delivering care. Unlike with consumer goods, a better product (healthcare decision) does not directly relate to increased demand or higher price. While this challenge may seem out of scope for technical AI innovation, it still constitutes a significant barrier to AI adoption compared to other industries.
EHR are an expensive problem. Many hospitals will have
10 or more different systems that all seem designed to be
inoperable with one another
An additional financial barrier comes from the current culture of software development. With the rapid growth of agile development, software development has become much more risk averse, with development teams searching for clear evidence of value. Increasingly, teams are working toward defining and rapidly deploying an MVP that can produce clear evidence of its beneficial impact in weeks. This is fine for retail companies that might want to try out new personalization approaches, where they can deploy and run A/B studies within a few weeks to collect evidence of the positive impact for their innovation. This seems to be out of reach for almost any AI healthcare system that focuses on high-risk clinician decision-making. Healthcare is unprepared for A/B testing, and we are not suggesting this would be a good thing. Our point is that new employees coming into software development bring a mindset that is in conflict with the pace AI in healthcare will need.
Unfortunately, very little research documents the details
of clinician decision-making and identifies the time and
place an AI inference might be experienced as most
valuable or explores which forms of AI output are most useful
The lack of a human-centered approach to AI innovation means that many innovation avenues are under-investigated. For example, little work has been done to apply techniques such as business process mining to healthcare. This would help reveal what the actual standard of care is, and clinicians could use this type of insight on their own behaviors to better understand and identify areas for improvement. As we mentioned previously, predicting events like expert disagreement empowers human decision-makers to reflect and consider before they have committed to a path of action. Healthcare practice has many goals; for example, rounding often mixes goals of patient care and health worker training. More human-learning focused AI systems might capture the dialog and provide feedback to an attending physician on the quality of their rounding – feedback such as waiting longer after asking a question and monitoring for implicit bias in who they ask questions of and who they compliment. Systems could also help with orchestration, the work of effectively coordinating work across the many experts. For example, in the ICU, a system might recommend an order for visiting patients during rounding based on the estimated time needed, the importance of early decision making for a patient (will they likely be extubated), and the physical layout of the rooms. There are many types of insights around human performance and processes that have been largely ignored by current technically focused research.
In this position paper, we elaborated on the interrelated challenges of feasibility, viability, and acceptance for moving clinical AI into the real world. These challenges will require thinking about not only the AI capabilities that are ripe for application, but also the business of healthcare, and the needs and desires of the frontline healthcare workers. We envision a future where researchers from AI, HCI, healthcare, design, and business research communities work together to take on these challenges. HCI and design researchers can focus on how technical advances in clinical AI might match to current needs and workflows of clinicians. AI and business researchers can work on low hanging fruit – worker needs and desires design research reveals that are likely to be solved with well known AI capabilities and existing healthcare data and infrastructure. We invite clinical AI researchers to advance collaborative research practices, effectively bridging the gap between research communities.
This material is based upon work supported by the National Science Foundation under Grant No. (2007501) and work supported by the National Institutes of Health (R35HL144804). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the National Institutes of Health.