Trust in AI healthcare technologies is often treated as an obtainable end-state enforceable by regulation, in which developers can claim their product to be 'trusted'. The article shows the limits of this approach, arguing instead for a processual understanding in which trust is understood to be dynamic and forever a state 'to come'. The argument is developed by considering several types of trust relations amongst key stakeholders in AI healthcare, including where developers often distrust users. Drawing on political theory and Coactive Design, the article argues that trust relations as a negotiation are integral to a well-functioning design process that not only supports the moral acceptability of AI healthcare technologies but also their innovation and efficacy.
The application of artificial intelligence technologies to assist human health carers is subject
to significant levels of hype – both positive claims it can save lives and cut costs, as well as
negative warnings that it risks a corporate ‘sell-out’ as hospitals become increasingly
dependent on large technological firms [
michael.strange@mau.se (M. Strange) 0000-0002-2903-7267 (M. Strange)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). paper seeks to advance understanding on the conditions needed for trust in AI healthcare technologies by considering trust as a relational negotiation requiring participation between various stakeholders, including patients, their families, healthcare professionals, technology developers and engineers, patient associations, civil society, as well as many less easily identified actors but who nonetheless impact that process. As will be argued, trust relations need to be more consciously conceived as part of the design process in AI healthcare. Discussions on trust in AI healthcare typically centre around how patients (also sometimes just referred to as the ‘public’, ‘consumers’, or ‘users’) and healthcare practitioners relate to the technology, with an implicit assumption that they need to be persuaded to accept and adopt AI healthcare products. How individuals come to (dis)trust technology is a complex negotiation between multiple factors, from their past experiences, expected benefits of the technology, the reputation of the producer but also immediate provider, and a host of other variables often only indirectly connected to the technology itself. The paper is structured as follows. It first provides an overview of the factors driving distrust in AI healthcare technologies. To develop a relational understanding of trust in AI healthcare, the paper then turns to critical theoretical approaches on both democracy and human rights that highlight the ways in which these ‘social goods’ are, despite common parlance, not fixed states or even end goals, but are constituted as processes that remain constant if they are to have normative value. The same applies to trust in AI, including within its healthcare application. That perspective will be further developed through connecting to the Coactive Design model. The article then concludes.
Healthcare is a sensitive field due to multiple factors including not only the obvious fact that it concerns life and death matters, but also the high cost of facilities coupled with often scarce resources in which technology fails have extremely serious and long-term consequences. Adopting a new technology like AI requires trust from a diverse range of stakeholders not limited to only healthcare professionals and patients. Trust impacts not only the willingness of actors to utilize AI but also how they experience the efficacy of those technologies. It is common to see trust in AI as concerning only how the technology is perceived by potential users, but this overlooks the wider relations in which the AI operates. If the context in which AI is both engineered and utilized is considered, that requires asking after a wider set of actors. For example, do developers trust users when they design AI? Are users seen as passive recipients or an integral part of a wider dynamic and ongoing relationship? In the field of healthcare, and for the purposes of simplicity, the paper considers four categories of actor: patients, healthcare professionals, society, and developers. These categories are simplifications but serve to illustrate the variety of trust relations at play within the implementation of AI technologies in healthcare.
Critical research on the impact of AI healthcare tools on patients covers several different
aspects coalescing around fear of discrimination and privacy violations. Obermeyer et al’s
[
The scale of data needed to train AI appears to necessitate mass transferability of
patient data, with examples being developed by the EU, for example, that promise security.
Yet, several recent and high-profile data breaches leave an open question as to whether such
security is possible. For example, hacks of healthcare data in the UK impacting blood banks
led to cancelled operations including heart surgery. Elsewhere in Europe, illicit access to
and exposure of data from a psychologist’s clinic led to black mail of former and present
patients faced with the threat of sensitive information being made public. In both cases, the
data breach occurred within a private firm sharing data with other healthcare
organisations. AI healthcare tools typically rely on private firms collaborating, and sharing
data, within a vast ecosystem of other actors that might not always be transparent to the
individual patient. This was a problem during the pandemic, for example, where the sudden
shift to online consultations meant some migrants in Europe no longer felt able to contact
their local doctors [
The relationship between healthcare professionals and the system in which they work puts them in a position of authority and responsibility, whilst also being required to adhere to clearly specified and evidence-based professional standards. A challenge for healthcare professionals trusting AI systems in their work is the ‘black box’ nature of the technology, where the outputs of large data models cannot be easily explained or understood. It is likely that the average doctor or nurse cannot explain the detailed workings of the pharmaceuticals and other more traditional technologies that are intrinsic to their work, yet authoritative individuals within their professional environment are able to do that on their behalf to the satisfaction of being evidence-based. It is often remarked that advancements in machine learning have moved beyond the understanding of all but those most actively engaged in the design of those systems. And, increasingly, the patterns AI can see within data are said to be undecipherable to even that small elite. Any obstacle to explainability brings risk within a sensitive field like healthcare as it makes it harder to ascertain acceptable standards of liability, including both premeditating against potential harms as well as allocating blame and punitive costs should harm occur.
A common solution proposed for mitigating risk of harm from AI, in healthcare but
also other sensitive contexts, is the so-called ‘human-in-the-loop’ in which a human
operative remains the ultimate decision-maker reviewing AI outputs at key action stages.
For example, whilst AI might propose a diagnosis, it is the human healthcare professional
that formally makes the diagnosis and on which any treatment and communication to the
patient is based. In that scenario, the AI is an information tool to be judged and utilized
within the framework of human expertise. Yet, it is far from clear that healthcare
professionals can remain active humans ‘in-the-loop’. First, the blackbox nature of AI
already discussed means healthcare professionals are being asked to take responsibility for
output they cannot themselves explain. When faced with scarce resources and severe time
constraints, doctors and nurses will be under pressure to reject or accept AI outputs without
having sufficient overview of their merits. Second, there is also concern of the ‘autopilot’
effect in which reliance on AI tools in areas of their work traditionally left to human
judgement will lead to a loss of their own abilities such that they have difficulties judging
correctly when to step in and take back control. This is what Reubin Binns has called the
‘human-in-the-loophole’ where the notion of a human supervisor is used to downplay risks
but, in practice, the humans involved feel unable to play that active role [
The design of healthcare systems varies globally with an obvious difference being the underlying funding model, including whether healthcare is provided on a ‘free-at-point-ofuse’ basis as well as the role of the private for-profit sector. The use of AI in healthcare has far reaching political-economic consequences due to both the need for large datasets that often transcend national borders, but also the dominant structure through which AI technologies are developed. That structure includes both large corporations that have a near-monopoly over the cloud computing facilities needed to process big datasets used for AI healthcare technologies, as well as a highly skewed geopolitical pattern in which the United States and China are global leaders with most national contexts left in the position of consumers only. Many societies are distanced from both development and ownership of the technologies and underlying infrastructure needed for healthcare systems to use AI. Whilst EU politicians and policy documents speak of a need to protect its ‘digital sovereignty’ by, for example, trying to develop data processing centres within the EU space that are controlled by EU-based interests, it is far from achieving that goal and, for much of the rest of the world, such a target is unobtainable.
Through processing sensitive data but also becoming the channel by which healthcare data becomes accessed, AI in healthcare will often function as an infrastructure supporting other parts of the healthcare system as opposed to operating as a standalone tool. Increasing usage will make it harder to discard within annual procurement rounds, and where specific AI healthcare technologies become pervasive as infrastructures within a hospital they are likely to impact what and how other healthcare technologies may be utilized in that context. The roll-out of AI in healthcare comes at the same time where there is heightened uncertainty over how to fund public healthcare systems in Europe, for example, with increasing usage of for-profit care firms. Many countries with state-funded healthcare systems have seen significant shifts in state-market relations in recent years. The roll-out of a technology that is dominated by for-profit corporations and that takes on an infrastructural role in healthcare systems cannot avoid being part of that wider politicaleconomic context. For society, AI healthcare technologies are therefore not neutral tools focused only on healthcare but relate to broader questions over how to best organize society. That is a fundamental and highly political question, greatly complicating society’s levels of (dis)trust towards AI in healthcare.
How patients, healthcare professionals, and society relate to AI healthcare technologies and
learn to (dis)trust those tools speaks to how users view the technology. However, trust
relations also concern how those building AI systems in healthcare view the users and wider
society. University courses for technology students will often include some aspect of
participatory design focused on, for example, how to engage and incorporate user feedback.
Yet, whilst feedback from users may be helpful, the rapid escalation in complexity that
comes with AI technologies coupled with the sensitivities when applied to healthcare means
there is a growing gap between the relative knowledge levels of developers and users. There
is also very limited space allocated for users and the wider public to enter the design stage.
Diversity within developer teams has dropped with fewer women getting to enter high-level
decision-making positions within technology firms [
Highlighted in the introduction, discussions on trust in AI healthcare are often guided by an implicit goal to persuade users – whether patients, healthcare professionals, or wider society – to accept and adopt AI healthcare products. That perspective treats trust as a oneway process, with AI developers (and owners) tasked to somehow convince users of the safety and efficacy of their products. However, this approach overlooks the complexity of trust as a complex and dynamic process that involves continuous negotiation between multiple stakeholders. Reframing trust as a negotiation requires recognizing that trust is not merely an outcome to be achieved but an integral part of the development and implementation of AI healthcare systems. Trust is shaped by a variety of factors, including past experiences, the perceived benefits of the technology, the reputation of the producer, and the credibility of the immediate provider. Moreover, trust is influenced by a host of other variables that are often only indirectly related to the technology itself, such as cultural values, social norms, and broader concerns. Central to this negotiation process is the space for dialogue between multiple stakeholders, which in healthcare includes not only patients but also their families, clinicians, device producers, as well as civil society such as patient associations.
Facilitating that negotiation is no easy task since it is shaped by two significant knowledge asymmetries between – first, healthcare professionals and patients; and second, AI developers and everyone else. The knowledge gap between healthcare professionals and patients is not new and, indeed, may also be said to give doctors and nurses the authority required to do their job. However, it does also create obstacles to communication and collaboration when that authority is questioned in times of uncertainty and rapid change, as we see with the sudden emergence of AI technologies. Given the challenges involved, then, is it possible to achieve the type of negotiation needed to build trust relations between key actors in AI healthcare, at what stage, and in what form? In response, the paper has no easy answers but offers some tentative thoughts by emphasizing the processual nature of negotiation and drawing on critical political theory.
Treating trust as a goal that can be obtained implies a point at which it has been obtained.
In other words, the goal of obtaining trust suggests a future statement in which one can
claim ‘our technology is trusted’. Jacques Derrida’s work on democracy is pertinent here,
taking a critical approach to claims that any nation-state is ‘democratic’ [
Network governance models in policymaking offer another useful framework for
understanding how trust can be fostered in the AI healthcare ecosystem, adding detail to
how one might build the kind of ‘trust to come’ approach suggested above. Network
governance speaks to the importance of involving diverse stakeholders in complex
governance processes, recognizing that effective policymaking requires input from a wide
range of perspectives. The concept of a democratic grammar of conduct [
There is overlap between seeing trust relations in AI healthcare as a negotiation and the
type of negotiation contained within Matthew Johnson’s model of ‘Coactive Design’ [
Critical political theory on democracy and human rights helps us better understand trust in AI healthcare through drawing out the processual character of trust as a normative concept. Like those other lofty political goals, if treated as an end-state the quest for trust in AI healthcare becomes limited to a questionable endeavour of persuasion in which developers seek to claim their product is the ‘most trustworthy’. In contrast, if we ask how developers conduct their relations with various stakeholders, including how open they are to hear societal concerns, interest turns not to trust as an end-state but, instead, as an ongoing and dynamic process. Rather than asking how to persuade patients and healthcare professionals to use AI, attention is placed on what kind of relations are needed for there to be trust in this highly sensitive domain.
Trust is also seen not just as a factor impacting whether people use technology, but as part of the ecosystem in which the technology is developed and operated. Asking how much developers trust users, for example, speaks to some of the significant barriers that prevent wider societal engagement and diversity within the design process. The fields of healthcare and technological development are not separate from the societies in which they operate, impacting and being impacted by wider political and economic issues. However, to be aware of those issues and how they impact the efficacy of AI technologies in healthcare requires design processes that do not limit trust relations until only the point of implementation. Instead, building trust relations in AI healthcare means using those relations as part of the design process.
A Coactive Design approach when combined with the processual approach to trust argued for in the paper positions negotiation as necessary in the everyday operation of AI healthcare technology. In other words, we see trust relations in AI healthcare as requiring negotiation throughout design, adoption, and usage of the technology. That is not a moral add-on but, as Johnson’s model shows, it is necessary to optimal functioning and efficacy of the technology.
Given the risks involved, healthcare is one of the most sensitive fields in which AI technology is being adopted, but also one in which there is considerable excitement with claims it will cut costs and improve outcomes by identifying conditions at points when treatment is cheaper and more effective, for example, as well as other benefits. Yet, such promise may well benefit only the already privileged depending on how AI is adopted in healthcare. To overcome potential worries that would slow the roll-out of AI healthcare technologies there is considerable focus on encouraging patients and healthcare professionals to trust it. The EU AI Act, for example, is part of that process to establish a basis for trust. However, as the paper argues, such efforts are mistaken where they treat trust as a fixed state that can be achieved through persuasion or regulation alone. Instead, the paper argues for a processual understanding of trust in which it is seen as a dynamic and ongoing negotiation requiring participation and collaboration between multiple stakeholders. Trust needs to be reframed as part of the design process, with trust relations empowering not only the moral acceptability of technology but as integral to innovation and, in healthcare, patient health and well-being.