Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 Towards Patient-oriented Transparency Short paper Dina Babushkina1[0000-0003-4899-8319] 1 University of Helsinki, Helsinki, Finland dina.babushkina@helsinki.fi Abstract. The paper introduces a patient-oriented concept of transparency in application to medical AI systems. It is argued that the patient’s perspective differs significantly from the user’s perspective and that there is a need to re- think what transparency entails for a patient rather than for a professional user. It is suggested that there are two major factors that influence the concept of transparency if we take the patient into account: (a) the active patient paradigm and (2) the patient’s trust in the medical sphere. It is further argued that, for an active patient, it is important to be able to make an informed decision about her health and voice criticism of a diagnosis/treatment, if she has sufficient ground for it. This poses one type of constraint on the concept of transparency. The other type of constraint derives from the fact that, in order to be trusted, medical expertise must be rooted in scientific methodology. Keywords: ethics of Artificial Intelligence, transparency, black box, patient moral rights, medical expertise 1 Accuracy vs. transparency? In his talk “AI and Trust: Explainability, Transparency” at Frankfurt Big Data Lab, Dragutin Petkovic posed a question to the audience, asking whether they would adopt a 99% accurate AI system which is essentially a black box, or a system that is less accurate (say in the range 85–95%) but explainable. At play here are a number of conflicting intuitions. One of these intuitions feeds on—in the absence of a better word—hope. Hope that there must be this one magic solution to problems that we either cannot solve or solve imperfectly. A kind a methodological panacea, if you wish. This solution should come from a source that is better, smarter, more perfect than we are. The other intuition takes the form of a critical, nagging voice telling us that something which cannot be explained should not be trusted. It also says that there are no magic solutions, and the best way to address a problem is through the careful analysis of causal chains. Both intuitions are understandable and relatable. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 117 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 But then, imagine going to a doctor. Say, you suffer from symptoms which are hard to diagnose. Your doctor uses an AI diagnostic system—let’s call it the BDSE, the “Best Diagnostic System Ever”—and, based on the results, explains that you have X. You find this hard to believe. You find yourself doubting the diagnosis, because it contradicts your experience and your own assessment of your symptoms (to the best of your knowledge). You ask for an explanation, and you are told that you should not doubt the results, because the BDSE is known for its very high degree of accuracy. Would you change your mind, based on this reply? Or, even better, should you? These are, however, two different questions. Whether you change your opinion is an idiosyncratic matter. You may do so for a number of reasons: because of a high level of trust in a professional medical opinion, because you cannot afford to pay for further examinations, or that you simply do not realize that disagreeing is an option. Whether you should be satisfied with the doctor’s reasoning and, as a result, silence your doubts, should be considered on different grounds. It is rather a questions of whether there is a well-justified reason for you to do so, such that it would be right for anyone in your shoes to accept. And how about the medical professional: should she accept the BDSE’s result just because it comes from a system with 99% accuracy, even when this result is not fully in sync with what she can observe herself and with what her patient reports? To make sure, the question being posed here is not whether “black-box” AI methods are useful, insightful, or whether we should use them. These questions are too broad. In many cases they are appropriate and we should use them. There is no objection, for example, when “black box systems” are used “as tools to inspire and guide human inquiry” and provided that, “as … any decision-making system, the black box … [is] used with knowledge, judgment, and responsibility” (Holm, 2019, 27). The question here, rather, concerns the ethical and rational constraints on their use. It is problematic to claim, for example, that black boxes should be used “when the cost of a wrong answer is low relative to the value of a correct answer” (Holm, 2019, 27). This is because the cost depends on the perspective. While, from the point of view of a medical institution, the cost of a low frequency mis-diagnosis might be low compared to the overall success of the AI-based diagnostic system, the cost is at a maximum for the person who is wrongly diagnosed and treated. The low-cost rational cannot be seen as a justifying reason1 for the use of black-box methodology. On the other hand, it is hard to deny that a black box may perform better than a human. For instance, “in reading standard field-of-view medical images, trained AI systems enhance the performance of human radiologists at detecting cancers. Although the cost of a wrong answer, whether a false negative or a false positive, may be high, 1 More on justificatory (normative) vs explanatory (motivating) reasons see, e.g., Smith (2000, 95). 118 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 the black box offers the best solution that is currently available” (Holm, 2019, 27). This might be true, but only in a strictly controlled environment where the data, on which the system performs its predictions, correctly represents the training dataset. Black-box systems do not operate with the concept of causality; they rely on pattern recognition. For this reason, the environment in which the patterns are detected must be meticulously checked. For that reason, we cannot accept the claim that a black box should be used “when it produces the best result” as a justifying reason either (Holm, 2019, 27). My point is that there are good reasons never to uncritically accept unexplainable AI systems when this may entail risk to health and well-being of medical patients. In what follows I will touch upon two such reasons: the patient’s active role in matters of her life and health and a patient’s trust in medical expertise. At the center of this discussion is the concept of transparency. By now it is apparent to many that transparency should be a requirement for AI systems (on transparency as a principle of Responsible Research and Innovation, see Dignum, 2019), especially in sensitive domains, when the output of such systems is used for decision making that affects well-being and poses a risk of harm. Whenever the need may arise to give an explanation for a decision, the possibility of explicating the role and contribution of an AI system must be provisioned. Transparency is an interesting and multi-dimensional topic. From the philosophical point of view, the first question concerns the kind of transparency needed and the nature of explanation itself (Coeckelbergh, 2020, 121). I want to bring in the perspective of the medical patient and see if this forces us to re- conceptualize transparency (on the importance of transparency in the medical sphere, see, e.g., Topol, 2019). Does the patient’s perspective pose specific constraints on the way AI systems should be transparent? If yes, then how are we to understand transparency in order for it to respect the interests and well-being of a patient? 2 Constraints on transparency My goal is to understand what the transparency requirement for AI systems in the medical sphere implies, when taking into account what matters for a patient. Why is this issue important? It is important because most of the discussion on the transparency of AI methods is focused on the user and, as a result, on the knowledge that a user must be granted access to (see, e.g., Samek et al., 2017; Hagras, 2018; Petkovic et al., 2018). But when it comes to medical application, the patient is often not the user. Their relation to a specific AI system (be it a diagnostic system, therapy system, or an advisor system) is radically different than that of a user. Medical personnel, administrators, researchers, and, perhaps, developers count as users, when they employ a system as a tool in their work or research. This constitutes professional use, the goal of which is to enhance and aid performance. Patients are on the receiving end of this professional use; they are either those who benefit or are harmed due to the use of the tool by medical professionals. (I will leave out of this discussion for now the use of health gadgets, when patients may also be considered as users in such cases.) 119 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 One could ask, Why should we care about the patient’s perspective? The patient’s perspective matters for several reasons. I will mention two: (1) Patients are those who are directly harmed in the case of an error. It is their life and health that is at stake, therefore it is only logical that their perspective is taken into account. (2) Transparency is not a simple binary phenomenon. When it comes to complex systems such as medical AI, the question of transparency is inseparable from the questions as to what must be revealed, to whom it is revealed, what their background is, and for which purpose they need the information. Different bits of information are important for creating understanding in a user, such as the radiologist or pathologist (whose goal it is to make sure that there is a correct diagnosis and treatment); an AI researcher (whose goal is to ensure that the system produces the result it is intended to produce); and a person whose lungs the machine is analyzing and who needs to decide how to take care of his health and well-being. Let me make a preliminary suggestion that two major factors influence the concept of transparency, if we take the patient into account: (1) The contemporary paradigm of participatory medicine and the concept of an “active patient,” and (2) The patient’s trust in medical expertise and the concept of good medical care (i.e., which practices must be considered acceptable as medical care). 3 Active patiency The contemporary paradigm of participatory medicine presupposes that the patient and medical personnel work together to establish the best way to safeguard the patient’s health.2 This dramatically changes the roles of the patient, from being a passive receiver of care (whose role in decision making is more or less reduced to obeying or disobeying the medical authority) to an active agent, carrying the responsibility for her own health. An active patient is delegated a much richer decision-making role. Such changes in patient responsibilities entail changes in the patient rights and capacities. Most importantly: 2 There are multiple sources on the topic, see, e.g., de Bronkart (2018), van der Eijk et al. (2013), Lejbkowicz et al. (2012), Hood and Auffray (2013, December 23). 120 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 (1) A right to make an informed decision in matters concerning her health and well- being. This presupposes that the patient must be able to make an informed judgment about her condition and state of health as well as the correctness of the diagnosis and effectiveness of treatment. (2) A right to oversee the diagnostic and treatment processes and challenge the decision concerning her health, if she has sufficient reason to do so. Neither of the patient’s rights, however, can be realized, if the process of medical diagnosis cannot be explained. I do not mean that a person should not be given freedom to prefer a more accurate system to the one that is more transparent. It is similar to the freedom to choose an alternative treatment if that is more rational in their own eyes. What I mean is that one should not expect that every patient will make this choice. Non- transparent systems should not become default medical tools that are used without a patient’s consent. In other words, acceptance of medical “black boxes” should not become a matter of the patient’s silent agreement. But what is it that matters from the patient’s perspective? If transparency is understood in terms of code availability, then it is not the issue. While the availability of the code may be crucial for research and development purposes, when it comes to the patient’s perspective, I do not think that making code known helps much. For one reason at least: it is almost impossible to read and interpret code without sufficient skills in programming. There are other technical senses of transparency, which again target professionals, such as users and policy makers. None of them, I believe, is directly relevant to the patient’s perspective. This is clear in the case of judicial transparency, which is understood as the requirement that “[a]ny involvement by an autonomous system in judicial decision-making should provide a satisfactory explanation auditable by a competent human authority“ (Petkovic, 2020). Another is failure transparency or the requirement that if an AI system caused harm, it should be possible to explain why (Petkovic, 2020). This is a rather narrow sense of transparency which demands an explanation post-factum, when the harm has already been done. What matters more for a patient is that there is a way to access the role of the system beforehand and prevent the harm. Such mistakes can become too costly for a patient. Transparency as explainability, or the ability to provide an explanation of the procedure that an AI followed and the specific decision that has been produced as a result, comes closer to what should represent a patient’s interests. But this is still problematic. To explain is not the same as being understood, and it is understanding that is crucial for the patient. One could, for instance, offer this explanation to a patient: the system used a pretrained deep neural network N to identify cancer and pneumonia patterns; its output was a set of values with corresponding percentages; the doctor based her diagnosis on the highest value output by the system. This explanation would constitute understanding (and contribute to a patient’s ability to critically evaluate this 121 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 result) only if the patient knows sufficiently well how such algorithms work, what their limitations are, and how to avoid misinterpreting the results. Most of us cannot do this. As a result, this explanation would not provide the right kind of knowledge for the patient to make an informed opinion. What matters for the patient is that she is given the right information that allows her to make an informed opinion such that she is in a position to challenge and object to an AI-based decision if she has sufficient reason to do so. And this bit is contained in the knowledge of the limitations of the AI method that influences the medical decision making in her case. Such limitations should ideally reflect how well the type of data that the model has been trained on represent the patient’s case, since this is one of the key parameters of the model’s accuracy (for a similar approach to data transparency see Yanisky-Ravid and Hallisey, 2018). To sum up, I suggest that the patient should have access at least to the following: · Information on which the AI method has been used for her diagnosis/treatment (so that she can refuse it or/and inform herself about the method from independent sources); · Explanation of how this specific method works and how to correctly interpret its output; · The limitations of this AI method. 4 Trust in medical expertise But what about a frequent argument that since “we routinely accept human conclusions without fully understanding their origin” (Holm, 2019, 27), we should also accept AI unexplainable results? Pointing out that people casually accept unsupported conclusions is not much of an argument. Again, they might do so for multiple reasons: lack of time, trust in authority, optimism, laziness, or carelessness. But these are idiosyncratic reasons, which do not oblige everyone to make the same choice. Observing how people behave tells us nearly nothing about what we should or have a good reason to do, and about what counts as a well-justified practice for everyone. Indeed, the conclusion one uncritically accepts may be based on bad reasoning and therefore may be mistaken. Furthermore, there is a difference between “I accept X even though I do not fully know why X” and “I accept X even though no one is able to explain why X.” While the first can still be justified in the case of well-established facts, the latter is esotericism, a mystical knowledge. In this light, there is another important issue to consider: the patient’s trust in medical expertise. This trust is crucial for the participatory medicine model to work. The foundation of this trust is the belief that medical institutions adhere to the principle of good medical and scientific practices. The patient has a good basis for trusting medical 122 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 advice, iff3 it is based on methods in line with established scientific standards. Among them is the requirement that a method or a tool is falsifiable, that is, there is a possibility of its refutation. It also means being aware and open about the imitations of the method, and the parameters that limit its applicability or challenge its use. These set additional scientific constraints on the concept of transparency. A transparent AI system is such that: · Its function is limited to that of a tool or method, aiding medical expert in her work; · The medical expert is able to scientifically access the tool/method and its limitations and correctly interpret its results; · The methods are such that they satisfy scientific constraints, and that implies that they are not obscure. The ability for a researcher to evaluate an AI method and for a doctor to understand how it works as a medical tool (and to which extent its results have to be supplemented by expert opinion) are necessary for the patient to be provided with the required level of transparency. For a medical specialist employing an AI tool, the first question should not be about accuracy but about adherence to scientific procedures. This would allow her to provide adequate support in building the patient’s trust. This, I must add, is not a problem of AI itself, but a much broader problem of the justifiability of a method. I do not mean that there is no room for anything that is not fully explainable or scientifically sound. Sometimes people may be justified in turning to alternative methods, especially when the medical system fails them for one reason or another. Such methods also rely on methods that they are unable to scientifically account for. But the fact remains that these are esoteric practices that are not supported by the scientific paradigm and they should remain a matter of personal discretion. One is free to use them, provided that one is aware of their limitations and risks. 5 References Coeckelbergh, M. (2020). AI ethics. Cambridge, MA: The MIT Press. de Bronkart, D. (2018). The patient’s voice in the emerging era of participatory medicine. The International Journal of Psychiatry in Medicine, 53(5–6), pp. 350-360. Dignum, V. (2019). Responsible Artificial Intelligence. Cham: Springer. Hagras, H. (2018). Toward Human-Understandable, Explainable AI. Computer, Vol. 51(9), pp. 28-36. 3 In standard logic notation, “iff” equals the biconditional “if and only if.” 123 Proceedings of the Conference on Technology Ethics 2020 - Tethics 2020 Holm, E. (2019). In Defence of the Back box. Science, Vol. 364(6435), pp. 26-27 Hood, L., Auffray, C. (2013, December 23). Participatory medicine: a driving force for revolutionizing healthcare. Genome Medicine. Retrieved from https://doi.org/10.1186/gm514. Lejbkowicz, I., Caspi, O., Miller, A. (2012). Participatory medicine and patient empowerment towards personalized healthcare in multiple sclerosis. Expert Review of Neurotherapeutics, 12: 3, pp.343-352. Petkovic, D. (2020, May 13). AI and Trust: Explainability, Transparency. [Online presentation]. Ethical Implications of AI and AI Tools Lab. http://www.bigdata.uni- frankfurt.de/data-challenge-ss-2020/ Petkovic, D., Altman, R., Wong, M., Vigil, A. (2018) Improving the Explainability of Random Forest classifier - User Centered Approach. Pac Symp Biocomput, 23, pp. 204‐215. Retrieved from https://pubmed.ncbi.nlm.nih.gov/29218882/ Samek, W., Wiegand, T., Müller, K. R. (2017). Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. Retrieved from ttps://arxiv.org/abs/1708.08296 Smith, M. (2000). The Moral Problem. Oxford: Blackwell. Topol, E.J. High-performance medicine: the convergence of human and artificial intelligence (2019). NatMed, 25, pp. 44-56. Retrieved from https://www.nature.com/articles/s41591-018-0300-7 van der Eijk, M, Nijhuis, F.A., Faber, M.J., Bloem, B.R. (2013). Moving from physician-centered care towards patient-centered care for Parkinson's disease patients. Parkinsonism Relat Disord. 19(11), pp. 923-927. Yanisky-Ravid, S., Hallisey, S. (2018). ‘Equality and Privacy by Design’: Ensuring Artificial Intelligence (AI) Is Properly Trained & Fed: A New Model of AI Data Transparency & Certification As Safe Harbor Procedures. SSRN Electronic Journal. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3278490 124