Explanation Ontology in Action: A Clinical Use-Case ? Shruthi Chari1[0000−0003−2946−7870] , Oshani Seneviratne1[0000−0001−8518−917X] , Daniel M. Gruen1[0000−0003−0155−3777] , Morgan A. 2[0000−0002−2739−5853] Foreman , Amar K. Das2[0000−0003−3556−0844] , and Deborah L. McGuinness1[0000−0001−7037−4567] 1 Rensselaer Polytechnic Institute, Troy, NY, USA {charis, senevo, gruend2}@rpi.edu,dlm@cs.rpi.edu 2 IBM Research, Cambrdige, MA, USA, 02142 morgan.foreman@ibm.com, amardasmdphd@gmail.com Abstract. We addressed the problem of a lack of semantic representa- tion for user-centric explanations and different explanation types in our Explanation Ontology (https://purl.org/heals/eo). Such a representation is increasingly necessary as explainability has become an important prob- lem in Artificial Intelligence with the emergence of complex methods and an uptake in high-precision and user-facing settings. In this submission, we provide step-by-step guidance for system designers to utilize our on- tology, introduced in our resource track paper, to plan and model for explanations during the design of their Artificial Intelligence systems. We also provide a detailed example with our utilization of this guidance in a clinical setting. Resource: https://tetherless-world.github.io/explanation-ontology Keywords: Modeling of Explanations and Explanation Types · Sup- porting Explanation Types in Clinical Reasoning · Tutorial for Explana- tion Ontology Usage 1 Introduction Explainable Artificial Intelligence (AI) has been gaining traction due to increas- ing adoption of AI techniques in high-precision settings. Consensus is lacking amongst AI developers on the type of explainability approaches and we observe a lack of infrastructure for user-centric explanations. User-centric explanations address a range of users’ questions, have different foci, and such variety provides an opportunity for end-users to interact with AI systems beyond just under- standing why system decisions were made. In our resource paper [2], we describe an Explanation Ontology (EO), which we believe is a step towards semantic en- coding of the components necessary to support user-centric explanations. This companion poster focuses on describing the usage steps (Section 2) that would ? Copyright c 2020 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). 2 S. Chari et al. serve as a guide for system developers hoping to use our ontology. We demon- strate the usage of the protocol (Section 3) as a means to support and encode explanations in a guideline-based clinical decision support setting. 2 Usage Directions for the Explanation Ontology Protocol 1 Usage of Explanation Ontology at System Design Time Inputs: A list of user questions, knowledge sources and AI methods Goal: Model explanations that need to be supported by a system based on inputs from user studies The protocol: 1. Gathering requirements (a) Conduct a user study to gather the user’s requirements of the system (b) Identify and list user questions to be addressed 2. Modeling (a) Align user questions to explanation types (b) Finalize explanations to be included in the system (c) Identify components to be filled in for each explanation type (d) Plan to populate slots for each explanation type desired (e) Use the structure of sufficiency conditions to encode the desired set of explanations System designers can follow the usage directions for EO at design time when planning for the capabilities of an AI system. The guidance aims to ensure that end-user requirements are translated into user-centric explanations. This pro- tocol guidance is supported by resources made available on our website. These resources include queries to competency questions to retrieve sample user ques- tions addressed by each explanation type3 and the components to be filled for each explanation type.4 Additionally, the sufficiency conditions that serve as a means for structuring content to fit the desired explanation type can be browsed via our explanation type details page.5 3 Clinical Use Case We applied the protocol to understand the need for explanations in a guideline- based care setting and identify which explanation types would be most relevant 3 https://tetherless-world.github.io/explanation-ontology/competencyquestions/ #question2 4 https://tetherless-world.github.io/explanation-ontology/competencyquestions/ #question3 5 https://tetherless-world.github.io/explanation-ontology/modeling/ #modelingexplanations Explanation Ontology in Action: A Clinical Use-Case 3 for this clinical use case. [2] Further, we utilized the EO to model some of the ex- planations we pre-populated into the system prototype. Hereafter, we describe our application of the EO usage guidelines and highlight how we used a user study to guide system design. As a part of the user study, we first held an expert panel interview to understand the clinicians’ needs when working with guideline-based care. We utilized the expert panel input to design a cognitive walkthrough of a clinical decision support system (CDSS) with some explana- tions pre-populated at design time to address questions that clinicians would want answered. Fig. 1. An overview of the user interface affordances we designed into the prototype to accommodate the various explanation types. The prototype of a CDSS we designed for the walkthrough, included al- lowances for explanations on different screens of the system (Fig. 1), in all, allowing clinicians to inspect a complicated type-2 diabetes patient case. Some examples of the pre-populated explanations are: contrastive explanations to help clinicians decide between drugs, trace based explanations to expose the guideline reasoning behind why a drug was suggested, and counterfactual explanations that were generated based on a combination of patient factors. Other examples were provided to us by clinicians during the walkthrough. Below, we step-through how the protocol guidance (Section 2) can be used to model an example of a counterfactual explanation from this clinical use case. A counterfactual explanation can be generated by modifying a factor on the treatment planning screen of the CDSS prototype. For illustration’s sake, let us suppose that this explanation is needed to address a question, “What if the patient had an ASCVD risk factor?" where the “ASCVD risk factor" is an alter- nate set of inputs the system had not previously considered. From our definition of a counterfactual explanation (https://tetherless-world.github.io/explanation- 4 S. Chari et al. ontology/modeling/#counterfactual), in response to the modification a system would need to generate a system recommendation based on the consideration of the new input. More specifically, in our use case, a system would need to consider the alternate input of the ASCVD risk factor in conjunction with the patient context and consult evidence from the American Diabetes Association (ADA) guidelines [5] to arrive at a new suitable drug recommendation for the patient case. With the alternate set of inputs and the corresponding recommendation in place, the counterfactual explanation components can now be populated as slots based on the sufficiency condition for this explanation type. A Turtle snippet of this counterfactual explanation example can be viewed in Fig. 2. Additionally, there have been some promising machine learning (ML) model efforts in the ex- plainability space [1,6] that could be used to generate system recommendations to populate specific explanation types. We are investigating how to integrate some of these AI methods with the semantic encoding of EO. Fig. 2. An annotated turtle representation of a counterfactual explanation that the CDSS would output in response to an additional factor the system had not considered. 4 Related Work In the explainable AI space, several research projects [4,7,3] have begun to ex- plore how to support end-user specific needs such as ours. Wang et al. [7] present a framework for aligning explainable AI practices with human reasoning ap- proaches, and they test this framework in a co-design exercise using a prototype CDSS. However, their framework isn’t in a machine-readable format and hence is difficult to reuse for supporting explanations in a system. The findings from their co-design process that clinicians seek different explanations in different sce- narios corroborate with those from our cognitive walkthrough, and EO can help support system designers in this endeavor. Similarly, Dragoni et al. [3] propose a rule-based explanation system in the behavior change space capable of pro- viding trace based explanations to end-users to encourage better lifestyle and Explanation Ontology in Action: A Clinical Use-Case 5 nutrition habits. While this explanation system was tested with real target users and subject matter experts, it is limited in scope by the types of explanations it provides. Vera et al. [4] have released a question bank of explanation question types that can be used to drive implementations such as ours. 5 Conclusion We have presented guidelines for using our Explanation Ontology in user-facing settings and have demonstrated the utility of this guidance in a clinical setting. This guidance describes an end-to-end process to support the translation of end- user requirements into explanations supported by AI systems. We are taking a two-pronged approach to pursue future work in operationalizing the use of our explanation ontology for clinical decision support systems. To this end, we are working towards building an explanation as a service that would leverage our explanation ontology and connect with AI methods to support the generation of components necessary to populate explanation types in different use cases. Further, we are implementing the clinical prototype as a functional UI with affordances for explanations. We expect the guidance along with open-sourced resources on our website will be useful to system designers looking to utilize our ontology to model and plan for explanations to include in their systems. Acknowledgments This work is done as part of the HEALS project, and is partially supported by IBM Research AI through the AI Horizons Network. References 1. Arya, V., Bellamy, R.K., Chen, P.Y., Dhurandhar, A., Hind, M., Hoffman, S.C., Houde, S., Liao, Q.V., Luss, R., Mojsilović, A., et al.: One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. arXiv preprint arXiv:1909.03012 (2019) 2. Chari, S., Seneviratne, O., Gruen, D.M., Foreman, M., Das, A.K., McGuinness, D.L.: Explanation Ontology: A Model of Explanations for User-Centered AI. In: Int. Semantic Web Conf. p. to appear. Springer (2020) 3. Dragoni, M., Donadello, I., Eccher, C.: Explainable ai meets persuasiveness: Trans- lating reasoning results into behavioral change advice. Artificial Intelligence in Medicine p. 101840 (2020) 4. Liao, Q.V., Gruen, D., Miller, S.: Questioning the AI: Informing Design Practices for Explainable AI User Experiences. arXiv preprint arXiv:2001.02478 (2020) 5. American Diabetes Assoc.: 9. Pharmacologic Approaches to Glycemic Treatment: Standards of Medical Care in Diabetes-2020. Diabetes Care 43(Suppl 1), S98 (2020) 6. Ribeiro, M.T., Singh, S., Guestrin, C.: Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016) 7. Wang, D., Yang, Q., Abdul, A., Lim, B.Y.: Designing theory-driven user-centric explainable AI. In: Proceedings of the 2019 CHI Conf. on Human Factors in Com- puting Systems. pp. 1–15 (2019)