The adoption of intelligent systems creates opportunities as well as challenges for medical work. On the positive side, intelligent systems have the potential to compute complex data from patients and generate automated diagnosis recommendations for doctors. However, medical professionals often perceive such systems as “black boxes” and, therefore, feel concerned about relying on system-generated results to make decisions. In this paper, we contribute to the ongoing discussion of explainable artificial intelligence (XAI) by exploring the concept of explanation from a human-centered perspective. We hypothesize that medical professionals would perceive a system as explainable if the system was designed to think and act like doctors. We report a preliminary interview study that collected six medical professionals' reflection of how they interact with data for diagnosis and treatment purposes. Our data reveals when and how doctors prioritize among various types of data as a central part of their diagnosis process. Based on these findings, we outline future directions regarding the design of XAI systems in the medical context.
• Human-centered computing ~ Interactive systems and tools • Human-centered computing ~ HCI design and evaluation methods Explainable artificial intelligence; human-centered design; medical data; system design.
ACM Reference format:
Intelligent systems, the computational agent that employs
algorithms to process and make sense of data, are becoming
increasingly ubiquitous in modern workplaces [
Copyright © 2019 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. promise of assisting human decision making through a data-driven approach, non-computing professionals often find it challenging to understand how the system transforms their initial input into a final decision and why.
In the medical field, systems such as
the CheXNet [
To help non-computing professional
Figure 1. The input and better comprehend results generated
output image of by intelligent systems, a growing body
CheXNet [
In this section, we first lay out a background review on XAI research, and then zoom into an HCI-oriented approach towards XAI. Since our focused field is in medicine, we further discuss prior work in medical AI, and specifically related to our interest— literature on the reasoning process of medical professionals.
Explainable artificial intelligence (XAI) raised a lot of concerns in
recent years [
A lot of work of interpretable machine learning has been done to
explain the inner principles of the machine learning models with
mathematical and algorithmic solutions [
In HCI, researchers are focusing on user’s interaction with the
intelligent systems and explanation is one important topic. HCI
researchers focus more on the interaction between the artificial
intelligent system and users and they have done a lot of work from
this aspect. Artificial intelligent systems have been criticized that
their rigid concepts are incompatible with human behavior styles
[
Systems need to provide users with not only results but also the
account of their behaviors [
In medical fields, artificial intelligent systems also have a broad
prospect. With the growth of availability of medical data and the
data processing techniques, artificial intelligent systems are
possible to be applied in the healthcare domain. They are able to
dig out useful information from a large amount of data which is
difficult to be processed by doctors and thus, assist the medical
decision making [
However, comparing to the booming industry, the actual usage of
the autodiagnostic system in hospitals is relatively low. A study
has been made to know doctors’ acceptance and the adopt
intention of these systems [
The explanation capabilities of artificial intelligence systems using
knowledge bases are firstly added for the applications in medical
decision making and computer-aided diagnosis in 1983 [
Cosby summarized tow models of clinical reasoning: analytical
and intuitive [
Also, due to the uniqueness of the medical field, medical
reasoning and decision-making mean more than what they mean in
other fields. From the doctors’ perspective, explanation of the
decision making process is not only how the results come out, but
also the cost of medical decisions such as the responsibility and
risk [
Broadly, the term ‘mental model’ is a concept derived from
cognitive psychology. It is the explanation of people’s thought
process about how things work [
Even though a lot of researches have been done to explain the intelligent systems. They seldom look into specific domains and incorporate empirical knowledge when explaining. We try to understand this problem from the doctors’ perspective and that’s why we seek to investigate the following research question: RQ: How do medical professionals interact with patients’ data for diagnosis and/or treatment purposes?
We conducted an interview study to explore research questions
presented above. Our current sample consists of six licensed
medical professionals working in California, United States. Each
interview lasts about 1 hour. During the research process, we
iterated between collecting new data, generating codes, and
revising/elaborating the existing coding book as suggested by the
grounded theory [
ID P1 P2 P3 P4 P5 P6
Male # of Years in the Medical Field 22 17 7 10 5 18 revised based on two pilot interviews with senior M.D. students at UCLA. The final protocol consisted of questions revolving around four issues: 1) the interviewee’s work and education experience in the medical field, 2) how s/he accesses to, processes and interprets medical related data during daily work practice, 3) challenges and solutions s/her ever experienced, if any, when working with medical data, and 4) experience and/or expectations of using computer-based systems to facilitate daily medical work. All interviews were conducted face-to-face in English and audio-taped for transcription.
Three authors of this paper analyzed the interview data together following an inductive approach. There were 60 codes and 201 quotations generated from the initial open coding. They yield participants’ self-reflection regarding the forms of data they interact with at daily medical work, the thinking process they go through when interacting with various data, the decisions they try to make based on data processing, and the types of work they have been delegating or hope to delegate to computer-based systems. We reiteratively discussed and compared between codes as they were generated. During the discussion, prioritization emerged as a focal theme from the data. It indicates that a central task medical professional performs during diagnosis is to prioritize among various and sometimes conflicting information given by patients, other doctors, and computer-based systems. We went through further coding to identify connections between this focal theme and other emerged themes and categories. The following section presents our detailed findings. Words and phrases directly quoted from participants are written in italic.
The process of generating a proper diagnosis and/or treatment plan is frequently described by our interviewees as being contextdependent, data-intensive, and open to alternative possibilities. In many cases, there lack one-to-one correspondences between signs, symptoms, and diseases. Medical professionals in the field, therefore, are often required to integrate various kinds of data and think outside the box. As it is pointed out by the following two participants: For medicine, it’s usually the grey area that matters. Everything is hardly black and white, and that’s why it is always difficult. … People say that medicine is both a science and an art, because every disease is different, and every patient’s representation will be different. Every doctor obviously has different steps in making the decision. [P6, Cardiologist] Authorities, like the American Heart Association, will publish guidelines and flow charts that we can refer. It prevents physicians from making ridiculous mistakes. But for more complex diseases, the guideline cannot include all of them. It will depend on the doctor’s experience or some innovations to accomplish the treatment. [P4, Family physician] In the rest of this section, we describe how medical professionals navigate around the complexity of their interaction with medical data. We identify three critical steps from interviewees’ reflection, including detecting/reacting to borderline cases, generating prioritization matrices, and coordinating with computer-based systems. Across all these steps, medical professionals keep prioritizing and re-prioritizing among information collected at different stages of the diagnosis process.
All participants of our interview reported running into borderline cases as the moments when the processing and interpreting of medical data turn challenging. One representative situation of encountering borderline cases is when the symptoms are still in their early state: At the very early state [of cancer], it is difficult to tell if the cell is abnormal. The architecture is minimally disrupted. You may think it is abnormal, but you don’t know whether it is malignance. We will show the cases to other colleges to get consents, or we have to say this case is inconclusive. [P1, Pathologist] In other situations of the borderline cases, medical professionals receive conflicting information that indicates different directions of the diagnosis: Many of us have run into cases when the MRI doesn’t confirm [our diagnosis]. We think the problem is in the right brain, but the image shows nothing there. In that case, we may do the test again. We can also go back to the patient to ask them again, or we discuss with other doctors. [P3, Neurologist] To clear up the ambiguity as indicated by the two quotations above, doctors often need to cross-validate their initial evaluation of the patient by requesting further data. Our interview with the six medical professionals documented multiple types of such data, including but not limited to, the patient’s demographic information, cardinal symptoms, results from further physical examinations and lab tests, historical data from reference groups, and evaluations given by other doctors.
Participants in our study yielded similar insights regarding how they deal with the rich yet complex medical data. Instead of following one hard rule of data processing, interviewees tend to weight/interpret each type of data differently based on their personalized prioritization matrices.
We identified six parameters from participants’ self-reflections that reveal how they perform data prioritization for diagnosis and/or treatment purpose. These parameters are labeled as below: Theoretical validity.
Severity of consequence.
Time constraint.
Domain of expertise.
Risk avoidance.
Technical feasibility.
Robustness of connections between signs, symptoms, and diseases as proved by theories, medical textbooks, and guidelines;
consequences if the detected signs/symptoms get put aside at this moment; side-effects of a treatment; interactions between different treatments;
symptoms connected to the doctor’s specialty; the level of confidence in offering a candidate treatment;
measurement; reliability of the technique; the false positive/negative rate of symptom detection.
Participants often used styles to describe the detailed prioritization matrices held by different doctors. Similar to other dispositional attributes such as personality, the prioritization matrix of a medical professional is perceived as being self-aware and consistent across various diagnosis made by the same individual: The diagnosis depends on many factors –severity, possibility, consistency with the patient’s history, and others. Some doctors will make the most severe issues on the priority, others will make the most possible ones their priority. It depends on their perspective. It also depends on the time concern. For example, neurologists may have a longer period of diagnosis, but surgeons and ER doctors don’t. [P2, Orthopedist] Some doctors trust images [over other information], like MRI, to tell what’s happening. About 80% of the time you would have good images. You are very confident about the diagnosis from the images. But I think most important information [to facilitate diagnosis] is what the patient tells you. It helps to track the patient’s history. [P6, Cardiologist] Our interviewees sometimes referred to the personalized prioritization matrices (or styles) to explain the disagreement between diagnosis suggestions provided by different doctors (see Figure 2 for illustration).
All medical professionals in our study reported that they have been using computer-based tools and systems to facilitate their daily work practice. Most participants, for instance, have greatly relied on cloud-based platforms to store and connect their local medical data with other databases [P1, P2, P4, P5, P6]. They also used various systems to generate automated calculation of chromosomes [P1], identify the degree of scoliosis [P2], check possible interactions between medications [P5], and etc. The primary function of such tools is to “provide quantified information to doctors, but not [to give] answers in terms of highlevel decisions [P3]”.
While participants were confident that the auto-quantified information given by systems is usually trustworthy and helpful, this optimism does not remain in their narratives of auto-diagnosis or treatment recommended by systems. The following quotation from P6 indicates a shared attitude as reflected across all the six interviews: There is a lot of advanced analysis involving machine learning, and some of them have entered the clinical realm. For example, you will have the nuclear images, and you will have the software telling you “it’s abnormal here and there.” It’s as if you have a second reader next to you. I would love to have the system generating results, but ultimately, it’s you that’s deciding on the diagnosis. When there is a disagreement, me and everyone will be overwriting the machine-generated interpretation. [P6, Cardiologist] To step forward from quantifying information to directly assisting diagnosis and treatment, systems are expected to “give an argument for why the data should be interpreted in that way [P5]”. The majority of our participants proposed the concept of reference and comparison as one approach to ground the systems’ diagnostic reasoning with that of human doctors’: Any machine has to give an evidence for the top reasons like in descending order for why in some matrices. It’s like if I say something and you think differently, then we should be able to really compare the two. Otherwise, it doesn’t matter if the machine’s suggestion is right. I don’t know what its thinking is and, ultimately, I take all the responsibility in this decision. [P1, Pathologist] There are different ways [to help validate the systems’ diagnosis recommendations]. One is showing me past examples in the database - will that support its conclusion? Another one is sources of data, something like research articles or convincing cases have been done. That’s upper-level evidence. [P5, General physician] Interviewees further suggested that to build an ideal autodiagnosis/treatment system, the algorithm should be able to contextualize its reference data with personalized information of a patient. Such contextualization work is what human doctors are good at based on their professional training and experience, but it is perceived to be the major obstacle for systems to overcome.
Based on the findings of the preliminary interview, we outline design suggestions for explainable medical AI systems. Specifically, we envisage a system that can Allow a medical professional to prioritize different types and sources of data by directly manipulating a user interface akin to our proposed prioritization matrix (Figure 2); Support gradual engagement of medical AI systems into a medical professional’s diagnosis process, spanning from low-level automated measurement tasks, to mid-level constraint-aware planning of medical tests, and to high-level suggestions of plausible diagnoses.
We thank Xiaohe Yang for assisting us to complete the interviews. We thank all the anonymous interviewees for their contributions to our study. We also thank Maie St. John, Peter Pellionisz and Jeff Liang for their valuable comments on earlier drafts of this paper.