Why these Explanations? Selecting Intelligibility Types for Explanation Goals Brian Y. Lim Qian Yang Ashraf Abdul, Danding Wang National University of Singapore Carnegie Mellon University National University of Singapore Singapore Pittsburgh, PA, USA Singapore brianlim@comp.nus.edu.sg yangqian@cmu.edu {ashrafabdul,wangdanding}@u.nus.edu ABSTRACT informing criminal justice. However, to ensure that we understand how these models and algorithms work, and to better The increasing ubiquity of artificial intelligence (AI) has spurred control them, these models need to be explainable. As a result, the development of explainable AI (XAI) to make AI more explainable AI research has been burgeoning with many understandable. Even as novel algorithms for explanation are algorithmic approaches being developed to explain AI and many being developed, researchers have called for more human HCI driven empirical studies to understand the impact of these interpretability. While empirical user studies can be conducted to explanations. We refer the interested reader to several literature evaluate explanation effectiveness, it remains unclear why reviews [1, 5, 10, 43]. specific explanations are helpful for understanding. We leverage To help end users to understand, trust, and effectively manage a recently developed conceptual framework for user-centric their intelligent partners, HCI and AI research have produced reasoned XAI that draws from foundational concepts in many user-centered, innovative algorithm visualizations, philosophy, cognitive psychology, and AI to identify pathways for interfaces and toolkits (e.g., [7, 20, 25, 36]). To make sense of the how user reasoning drives XAI needs. We identified targeted variety of explanations, several explanation frameworks have strategies for applying XAI facilities to improve understanding, been proposed for knowledge-based systems [10], recommender trust and decision performance. We discuss how our framework systems [12], case-based reasoning [39], intelligent decision aids can be extended and applied to other domains that need user- [40], tutoring systems [10], intelligible context-aware systems centric XAI. This position paper seeks to promote the design of [24], etc. These frameworks are mostly taxonomic or driven by XAI features based on human reasoning needs. clearly defined principles (e.g. [21]). In this work, we aim to identify theories in human thinking that drives the needs for CCS CONCEPTS different types of explanations. • Human-centered computing ~ Human computer interaction Indeed, some work has drawn from more formal theories. Recent writings by Miller, Hoffman and Klein discussed relevant KEYWORDS theories from philosophy, cognitive psychology, social science, Intelligibility; Explanations; Explainable artificial intelligence; and AI to inform the design of eXplainable AI (XAI) [13, 14, 15, 19, Decision making 31]. Miller noted that much of XAI research tended to use the researchers’ intuition of what constitutes a “good” explanation. ACM Reference format: He argued that to make XAI usable, it is important to draw from Brian Y. Lim, Qian Yang, Ashraf Abdul and Danding Wang. 2019. Why social sciences. Hoffman et al. [13, 14, 15] and Klein [19] these Explanations? Selecting Intelligibility Types for Explanation Goals. summarized several theoretical foundations of how people In Joint Proceedings of the ACM IUI 2019 Workshops, Los Angeles, USA, formulate and accept explanations, empirically identified several March 20, 2019, 7 pages. https://doi.org/10.1145/1234567890 purposes and patterns for causal reasoning, and proposed ways that users can generate self-explanations to answer contrastive questions. However, it is not clear how best to operationalize this 1 Introduction rich body of work in the context of XAI-based decision support The recent success of artificial intelligence (AI) is driving its systems for specific user reasoning goals. Hence, adding on to this prevalence and pervasiveness in many domains of decision line of inquiry, we have recently proposed a theory-driven, user- making from supporting healthcare intervention decisions to centric XAI framework that connects XAI explanation features to underlying reasoning processes that users have for explanations [42]. Drawing on this framework, XAI researchers and designers IUI Workshops'19, March 20, 2019, Los Angeles, USA. can identify pathways along which human cognitive patterns Copyright © 2019 for the individual papers by the papers' authors. drives needs for building XAI. By articulating a detailed design Copying permitted for private and academic purposes. This volume space of technical features of XAI and user requirements of is published and copyrighted by its editors. human reasoning, we intend that our framework will help ExSS '19, March 20, 2019, Los Angeles, CA, USA. B. Y. Lim et al. Understanding People informs Explaining AI How People should Reason and Explain How XAI Generates Explanations • Explanation goals • Bayesian probability filter causes | generalize and learn | predict and control | prior | conditional | posterior transparency | improve decisions | debug model | moderate trust • Similarity modeling clustering | classification | dimensionality reduction | rule boundaries • Inquiry and reasoning • Intelligibility queries | induction | analogy | deduction abduction | hypothetico-deductive model inputs | outputs | certainty | why | why not | how to | what if | when • Causal explanation and causal attribution | • XAI elements contrastive | counterfactual | transfactual | attribution name | value | attribution | clause | instance Figure 1. Partial Conceptual framework for Reasoned Explanations (of [42]) that describes how human reasoning processes (left) informs XAI techniques (right). Points describe different theories of reasoning, XAI techniques, and strategies for designing XAI. Arrows indicate pathway connections: red arrows for how theories of human reasoning inform XAI features, and grey for inter-relations between different reasoning processes and associations between XAI features. Only some example pathways are shown. For example, to find the cause of an application behavior, user could seek a contrastive explanation of counterfactuals to filter causes (grey arrow); this can be supported with why not and how to explanations, respectively (red arrows). To help users generalize and learn about the application behavior, we should support reasoning processes (grey arrow) of induction, analogy and deduction by highlighting similarity/differences, various forms of probability and rule boundaries respectively (red arrows). developers build more user-centric explainable AI-based systems of different XAI techniques [1] and considered what reasoning or with targeted XAI features. cognitive theories would justify the need for such methods. In We have previously introduced the conceptual framework in addition, we considered how people reason and make decisions so [42] and we refer interested readers to study the original work. In as to design XAI that normalizes to their thought processes; this this position paper, we further demonstrate how to use the minimizes the learning curve of XAI facilities. Our literature framework to design targeted explanation features, focusing on search was inspired by work from Miller [31], but we limited our the choice of explanation types defined by Lim and Dey [24, 25]. scope to philosophy, cognitive psychology, and AI. We iteratively Lim and Dey defined their explanation taxonomy for context- refined our framework by 1) finding concepts in XAI, reasoning aware computing and described how users might use them [24] as and psychology, 2) drawing connections between them to well as empirically demonstrated the effectiveness of some elucidate relationships, 3) finding gaps to justify why certain XAI explanation types [23]. However, in later studies, they found that techniques could be useful, and 4) searching for more concepts. users could also reason differently than anticipated and have We have developed a conceptual framework that links different preferences for explanation types even for the same concepts in human reasoning processes with explainable AI tasks [27, 29]. For example, to understand how a mobile app would techniques. By considering two aspects of the human and the behave in a new situation, Lim and Dey found that users could machine, we further divide the framework into four main either use the What If explanation, or exercise the actual scenario modules. We focus on two modules here. First, we identify how and inspect the Why explanation. people ideally reason and why we seek explanations (2.1). These Our conceptual framework in [42] provides theoretical support articulate reasoning methods and explanation types that provide for why users preferred certain explanations in addition to also the foundation of what good decision aids should support. Second, describing how people are subject to reasoning fallacies and we describe various AI modeling and XAI facilities, and cognitive biases as well as how to select explanations to mitigate contextualize how they have been developed to support certain these biases. In this work, we focus on supporting three reasoning methods (2.2). Figure 1 shows these key modules and explanation goals by providing examples for specific XAI features pathways linking them to illustrate how some reasoning methods that support them and relate them to Lim and Dey's intelligibility can be supported by XAI facilities. types. To provide some context before we apply the framework, we summarize relevant parts of the original work [42] in the next 2.1 How People should Reason and Explain section. This section informs how XAI can support different explanation types by articulating how people understand events or observations through explanations. We drew these insights from 2 XAI Framework of Reasoned Explanations the fields of philosophy, and cognitive psychology, specifically 1) We performed a literature review and synthesized a conceptual different ways of knowing, 2) what structures contain knowledge, framework from rationalizing logical connections. Rather than 3) how to reason logically and 4) why we seek explanations. perform a comprehensive encyclopedic literature review of 2.1.1 Explanation Goals. The needs for explanations are relevant concepts in XAI [1, 5, 10, 43], our goal was to create an triggered by a deviation from expected behavior [31], such as a operational framework with which developers of XAI interfaces curious, inconsistent, discrepant or anomalous event. and systems can use. We started with an existing literature review 2 Why these Explanations? ExSS '19, March 20, 2019, Los Angeles, CA, USA. Selecting Intelligibility Types for Explanation Goals Alternatively, users may also seek to monitor for an expected, 2.1.3 Causal Attribution and Explanations. As users inquire for important or costly event. Miller identified that the main reason more information to understand an observation, they may seek why people want explanations is to facilitate learning by allowing different types of explanations. Miller identified causal the user to (i) filter to a small set of causes to simplify their explanations as a key type of explanation, but also distinguished observation, and to (ii) generalize these observations into a them from causal attribution, and non-causal explanations [31]. conceptual model where they can predict and control future Causal attribution refers to the articulation of internal or phenomena [31]. The latter goal of prediction is also described as external factors that could be attributed to influence the outcome human-simulatability [30]. We orient our discussion of or observation [11]. Miller argues that this is not strictly a causal explanations with respect to these broad goals of finding causes explanation, since it does not precisely identify key causes. and concept generalization. Nevertheless, they provide broad information from which users From the AI research perspective, a recent review by Nunes can judge and identify potential causes. Combining attribution and Jannach summarized several purposes for explanations [32]. across time and sequence would lead to a causal chain, which is Explanations are provided to support transparency, where users sometimes considered a trace explanation or line of reasoning. can see some aspects of the inner state or functionality of the AI Causal explanation refers to an explanation that is focused on system. When AI is used as a decision aid, users would seek to use the selected causes relevant to interpreting the observation with explanations to improve their decision making. If the system respect to existing knowledge. This requires that the explanation behaved unexpectedly or erroneously, users would want be contrastive between a fact (what happened) and a foil (what is explanations for scrutability and debugging to be able to identify expected or plausible to happen). Users can ask why not to the offending fault and take control to make corrections. Indeed, understand why a foil did not happen. The selected subset of this goal is very important and has been well studied regarding causes thus provides a counterfactual explanation of what needs user models [3, 16] and debugging intelligent agents [21]. Finally, to change for the alternative outcome to happen. This helps explanations are often proposed to improve trust in the system people to identify causes, on the scientific basis that manipulating and specifically moderate trust to an appropriate level [4, 6, 26]. a cause will change the effect. This also provides a more usable 2.1.2 Inquiry and Reasoning. With the various goals of explanation than causal attribution, because it presents fewer explanations, the user would then seek to find causes or generalize factors (reduces information overload) and can provide users with their knowledge and reason about the information or a greater perception of control, i.e., how to control the system. A explanations received. Pierce defined three kinds of inferences similar method is to ask what if the factors were different, then [34]: deduction, induction, and abduction. Deductive reasoning what the effect would be. Since this asks about prospective future “top-down logic” is the process of reasoning from premises to a behavior, Hoffman and Klein calls this transfactual reasoning; conclusion. Inductive reasoning “bottom-up logic” is the reverse conversely, counterfactual reasoning asks retrospectively [13, 14]. process of reasoning from a single observation or instance to a This articulation highlights the importance of contrastive (Why probable explanation or generalization. Abductive reasoning is Not) and counterfactual (How To) explanations instead of simple also the reverse of deductive reasoning and reasons from an trace or attribution explanations typically used for transparency. observation to the most likely explanation. This is also known as 2.1.4 Summary. We have identified different inquiry and “inference to the best explanation”. It is more selective than explanation goals, rational methods for reasoning, causal and inductive reasoning, since it prioritizes hypotheses. non-causal explanation types, and evaluation with decisions to Popper combined these reasoning forms into the Hypothetico- describe a chain of reasoning that people make. We next describe Deductive model as a description of the scientific method [2, 35]. various explanations and AI facilities and how they support The model describes the steps of inquiry as (1) observe and reasoning. identify a new problem, (2) form a hypothesis as induction from observations, (3) deduce consequent predictions from the 2.2 How XAI Generates Explanations hypotheses, and (4) test (run experiments) or look for (or fail to Now we turn to how algorithms generate explanations, in find) further observations that falsify the hypotheses. It is searching for connections with human explanation facilities. We commonly used and taught in medical reasoning [8, 9, 33]. A key characterize AI and XAI techniques by how they (1) semantically aspect of the HD model is hypothesis generation where support human reasoning specific methods of scientific inquiry, observation of the current state can help the user decide whether such as Bayesian probability, similarity modeling, and queries; to test for relationships between potential causes and the outcome and (2) how to represent explanations with visualization methods, effect. data structures and atomic elements. Where relevant we link AI Finally, analogical reasoning is the process of reasoning from techniques back to concepts (green text) in rational reasoning. one instance to another. It is a weak form of inductive reasoning Bold text refers to key constructs in each module in the since only one instance is considered instead of many examples framework, and italic text refers to sub-constructs. [41]. Nevertheless, it is often used in case base reasoning and in 2.2.1 Bayesian Probability. Due to the stochastic nature of legal reasoning to explain based on precedence (same case) or events, reasoning with probability and statistics is important in analogy (similar case) [22]. decision making. People use inductive reasoning to infer events and test hypotheses. Particularly influential is Bayes theorem that 3 ExSS '19, March 20, 2019, Los Angeles, CA, USA. B. Y. Lim et al. describes how the probability of an event depends on prior 3 Intelligibility Types knowledge of observed conditions. This covers specific concepts We employ the taxonomy of Lim and Dey [24, 28] due to its of prior and posterior probabilities, and likelihood. Understanding pragmatic usefulness to operationalize in applications and to outcome probabilities can inform users about the expected utility. leverage the Intelligibility Toolkit [25] that makes it convenient Bayesian reasoning helps decision makers to reason by noting to implement a wide range of explanations. While it does not the prevalence of events. E.g., doctors should not quickly conclude currently generate recent state-of-the-art explanations and that a rare disease is probable, and they would be interested to models, the explanation data structures allow it to be extended to know how influential a factor or feature is to a decision outcome. support feature attribution and rules explanations. We reapply the 2.2.2 Similarity Modeling. As people learn general concepts, original definitions to more general applications of machine they seek to group similar objects and identify distinguishing learning beyond context-aware systems and introduce new types. features to differentiate between objects. Several classes of AI We also describe and situate the explanation types in context of approaches have been developed, including modeling similarity underlying reasoning processes. with distance-based methods (e.g., case base reasoning, clustering Inputs explanations inform users what input values from data models), classification into different kinds (e.g., supervised instances or sensors that the application is reasoning for the models, nearest neighbors), and dimensionality reduction to find current case. When a user asks a why question, she may naively latent relationships (e.g., collaborative filtering, principal be asking for the Inputs state. We also consider this to be the basic components analysis, matrix factorization, and autoencoders). form of explanation to support transparency by showing the Many of these methods are data-driven to match candidate objects current measured input or internal state of the application. with previously seen data (training set), where characterization What Output explanations inform users what is the current depends on the features engineered and the model which frames outcome, inference, or prediction and what possible output an assumed structure of the concepts. Explanations of these options the application can produce. For applications that can mechanism are driven by inductive and analogical reasoning to have different outcome values (multiclass or multilabel), we can understand why certain objects are considered similar or also show Outputs explanations. This lets users know what it can different. Identifying causal attributions can then help users do or what states it can be in (e.g., activity recognized as one of ascertain the potential causes for the matching and grouping. three options: sitting, standing, walking). This helps users Note that while rules appear to be a distinct explanation type, we understand the extent of the application’s capabilities. could consider them as descriptions of the boundary conditions Certainty explanations inform users how (un)certain the between dissimilar groups. application is of the output value produced. They help the user 2.2.3 Intelligibility Queries. Lim and Dey identified several determine how much to trust the output value and whether to queries (called intelligibility queries) that a user may ask of a consider an alternative outcome. While originally, Lim and Dey smart system [24, 25]. Starting from a usability-centric considered the confidence outcome of a predictive model, this can perspective, the authors developed a suite of colloquial questions now include stochastic uncertainty from Bayesian modeling about the system state (Inputs, What Output, What Else Outputs, approaches, which is essentially the posterior probability. Certainty), and inference mechanism (Why, Why Not, What If, Furthermore, we have found that users may reason with prior and How To). While they initially found that Why and Why Not conditional probability, so three types of uncertainty should be explanations were most effective in promoting system supported: prior, conditional, posterior. understanding and trust [23], they later found that users may Why explanations inform users why the application derived exploit different strategies to check model behavior and thus use its output value from the current (or previous) input values. This different intelligibility queries for the same interpretability goals is typically represented as a set of triggered rules (rule trace) for [26, 29]. rule-based systems or feature attributions (or weights of 2.2.4 XAI Elements. We identify several building blocks that evidences) for why the inferred value was inferred over compose many XAI explanations. By identifying these elements, alternative values. Compared to Input explanations, Why we can determine if an explanation strategy has covered explanations focus on highlighting a subset of key variables or information that could provide key or useful information to users. clauses, though this does not specifically support counterfactual This reveals how some explanations are just reformulations of the reasoning, especially for multi-class classification systems. same explanation types but with different representations, such Why Not explanations inform users why an alternative that the information provided and interpretability may be similar. outcome (foil) was not produced, with respect to the inferred Currently, showing feature attribution or influence is very outcome (fact), given the current input values. Why Not popular, but this only indicates which input feature of a model is explanations provide a pairwise comparison between the inferred important or whether it had positive or negative influence outcome and an alternative outcome. Similar to Why explanations towards an outcome. Other important elements include the name that help users to focus on key inputs, Why Not explanations and value of input or outputs (generally shown by default in focus on salient inputs that matter for contrasting between the explanations, but fundamental to transparency), and the clause to fact and foil. With the fewer features highlighted, this can support describe if the value of a feature is above or below a threshold (i.e., counterfactual reasoning, where the user learns how to change a rule). key input values to achieve the alternative outcome. Hence, such 4 Why these Explanations? ExSS '19, March 20, 2019, Los Angeles, CA, USA. Selecting Intelligibility Types for Explanation Goals Why Not explanations are essentially How To explanations. Note We identified three pathways to help users narrow down and that we can also interpret a Why explanation as a contrast identify specific causes for a particular system outcome (see between the inferred outcome and all other alternative outcomes. Figure 2). While Input explanations are most basic and However, also note that Why Not explanations generated as colloquially queried by users, we identify that users would inspect feature attribution or weights of evidence are not particularly the input feature values to find anomalies, discrepancies, or useful for How To explanations. As with Lim and Dey, we note surprising values, then generate hypotheses for what could be that Why Not explanations are important to support, since users wrong. This is not particularly efficient, since users are not would typically ask for explanations when something unexpected directed to any salient cause, but it can allow users to determine happens, i.e., they expect the foil to happen. This agrees with their own hypotheses for causes. Going even further and giving Miller that most users truly ask for contrastive explanations (Why users more choice for hypothesis generation, we can support the Not) and that these should be explained with counterfactuals discovery of latent factors. While not originally defined in the (How To) [31]. Intelligibility framework [24, 25, 28], recent work by Kim et al. on What If explanations allow users to anticipate or simulate TCAV [18] allows users to specify their own concepts of interest what the application will do given a set of user-set input values. and test if they are influential in a model’s inference. While this straightforward explanation type has received little The second pathway involves showing Why explanations as attention in recent AI research, it is an intuitive technique to either a rule trace or feature attribution (or importance). This is support human simulatability defined by Lipton [30] and supports driven by the users identifying the influence or attribution due to transfactual reasoning defined by Hoffman and Klein [13]. various causes (features) or by tracing deductive paths in the When explanations (new) indicate under what circumstance or system rule logic. scenario, or with what instance case would a particular outcome The third pathway involves contrasting the inferred outcome happen. This can be used to explain for inferred or alternative (fact) with the expected outcome (foil). Salient large feature outcomes. Unlike Why or Why Not explanations which focus on attribution differences can call the user’s attention to potential input feature attributions or values, this focuses on the instance causal features, but rules provide a more actionable method that entity as a whole. Thus, it is suitable for exemplar, prototype or explains how specific feature values could have led to the case-based explanations. Unlike How To explanations, this does counterfactual case outcome. not describe counterfactuals of a subset o inputs to change a scenario to have a different outcome. Note that we use a different 4.2 Generalize and Learn definition as originally defined by Lim [28], which referred to the timestamp of the inference event. 4 Selecting Intelligibility for Explanation Goals We had previously summarized several goals or reasons why Figure 3. Pathways for using Certainty and When explanations to people ask for explanations. These are primarily to improve their support users to generalize and learn about how the system would understanding of the AI-driven application, or the situation, or to behave for similar cases. improve their current or future ability to act predictably and correctly. In this section, we describe how to support three We identified two main pathways to help the user learn a general explanation goals — filter causes, generalize and learn, and predict mental model of how the system would behave (see Figure 3). The and control — with the Intelligibility explanation types. By first pathway involves reasoning by induction, the user could be relating the use of these explanations explicitly back to user goals, interested to know the likelihood of the outcome i) in general we identify pathways to justify the use of various explanations in (overall), ii) the system’s confidence or certainty of the outcome explainable AI. While our text and Figure 1 already describe some prediction for the current instance, or iii) an intermediate pathways, we articulate these clearer in this section. certainty where only some features matter (e.g., disease risk for all males, given that a patient is male). 4.1 Find and Filter Causes The second pathway involves simpler but narrower reasoning by analogy, where the user looks at one instance at a time to form a detailed understanding of similar specific cases. Here, the system proposes the examples, such as i) prototypes to indicate median instances for each outcome type, ii) critique examples to indicate examples of a desired outcome that are close to the decision boundary [17], or iii) counter-examples that are similar Figure 2. Pathways for using Inputs, Why and Why Not to the current instance but have different predicted outcomes [38]. explanations to help the user to find and filter causes for the current system inference behavior. 5 ExSS '19, March 20, 2019, Los Angeles, CA, USA. B. Y. Lim et al. 4.3 Predict and Control [4] Antifakos, S., Schwaninger, A., & Schiele, B. (2004, September). Evaluating the effects of displaying uncertainty in context-aware applications. In International Conference on Ubiquitous Computing (pp. 54-69). Springer, Berlin, Heidelberg. [5] Biran, O., & Cotton, C. (2017). Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI) (p. 8). Figure 4. Pathways for using How To and What If explanations to [6] Bussone, A., Stumpf, S., & O'Sullivan, D. (2015, October). The role of support users to predict how the system would behave in a future explanations on trust and reliance in clinical decision support systems. In Healthcare Informatics (ICHI), 2015 International Conference on (pp. 160- case and to control its behavior for current or similar cases. 169). IEEE. [7] Coppers, S., Van den Bergh, J., Luyten, K., Coninx, K., van der Lek-Ciudin, I., Vanallemeersch, T., & Vandeghinste, V. (2018, April). Intellingo: An We identified three pathways to help users to predict the system’s Intelligible Translation Environment. In Proceedings of the 2018 CHI future behavior and control current behavior (see Figure 4). First, Conference on Human Factors in Computing Systems (p. 524). ACM. using deductive reasoning, users can read the full rule-set of the [8] Croskerry, P. (2009a). A universal model of diagnostic reasoning. Academic medicine, 84(8), 1022-1028. general How To explanation to predict how the system would [9] Elstein, A. S., Shulman, L. S., & Sprafka, S. A. (1978). Medical Problem Solving: inference. However, this can be tedious for a large rule-set. An Analysis of Clinical Reasoning. Cambridge, MA: Harvard University Press. [10] Graesser, A.C., Person, N., Huber, J. (1992). Mechanisms that generate Second, focusing on a specific contrast case, users can use the questions. In: Lauer, T.W., Peacock, E., Graesser, A.C. (Eds.), Questions and How To counterfactual explanations of rule-based Why Not Information Systems. Lawrence Erlbaum, Hillsdale, NJ, pp. 167–187. explanations to understand how they could try to change the [11] Heider, F. (2013). The psychology of interpersonal relations. Psychology Press. [12] Herlocker, J., Konstan, J., and Riedl, J. (2000). Explaining collaborative filtering situation for a different outcome. Anchors by Ribeiro et al. provide recommendations. In Proceedings of the 2000 ACM conference on Computer a good recent method for counterfactual explanations to support supported cooperative work (CSCW'00). ACM, New York, NY, USA, 241-250. [13] Hoffman, R. R., & Klein, G. (2017a). Explaining explanation, part 1: theoretical How To explanations [37]. Third, users could use a What If foundations. IEEE Intelligent Systems, (3), 68-73. explanation to test specific instances that they are interested in; [14] Hoffman, R. R., Mueller, S. T., & Klein, G. (2017b). Explaining Explanation, i.e., they set input values and observe the simulated outcomes. Part 2: Empirical Foundations. IEEE Intelligent Systems, 32(4), 78-86. [15] Hoffman, R., Miller, T., Mueller, S. T., Klein, G., & Clancey, W. J. (2018). This is similar to When explanations, but the user chooses the Explaining Explanation, Part 4: A Deep Dive on Deep Nets. IEEE Intelligent input states and example. Systems, 33(3), 87-95. [16] Kay, J. (2001). Learner control. User modeling and user-adapted Note that we consider explanations that build tree explainer interaction, 11(1-2), 111-127. models to be equivalent to rule-based explanations, since we can [17] Kim, B., Khanna, R., & Koyejo, O. O. (2016). Examples are not enough, learn use first-order logic to transform them [25, 28]. Furthermore, we to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems (pp. 2280-2288). do not know of any feature attribution-based explanations that [18] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., & Viegas, F. (2018, July). can specifically satisfy the explanation goal of prediction and Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International Conference on Machine control. The popularity of feature attributions presents a big gap Learning (pp. 2673-2682). in the research in XAI which tend to not produce actionable [19] Klein, G. (2018). Explaining Explanation, Part 3: The Causal Landscape. IEEE explanations. Intelligent Systems, 33(2), 83-88. [20] Krause, J., Perer, A., & Ng, K. (2016, May). Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5686-5697). 5 Conclusion ACM. [21] Kulesza, T., Burnett, M., Wong, W. K., & Stumpf, S. (2015, March). Principles We have described a theory-driven conceptual framework for of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th international conference on intelligent user designing explainable facilities by drawing from philosophy, interfaces (pp. 126-137). ACM. cognitive psychology and artificial intelligence (AI) to develop [22] Lamond, G. (2006). Precedent and analogy in legal reasoning. The Stanford user-centric explainable AI (XAI). Using this framework, we can Encyclopedia of Philosophy. https://plato.stanford.edu/entries/legal-reas- prec/. Retrieved 10 September 2018. identify specific pathways for how some explanations can be [23] Lim, B. Y., Dey, A. K., & Avrahami, D. (2009a, April). Why and why not useful, how certain reasoning methods fail due to cognitive biases, explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing and how to apply different elements of XAI to mitigate these Systems (pp. 2119-2128). ACM. failures. By articulating a detailed design space of technical [24] Lim, B. Y., & Dey, A. K. (2009b, September). Assessing demand for features of XAI and connecting them with user requirements of intelligibility in context-aware applications. In Proceedings of the 11th international conference on Ubiquitous computing (pp. 195-204). ACM. human reasoning, our framework aims to helps developers build [25] Lim, B. Y., & Dey, A. K. (2010, September). Toolkit to support intelligibility in more user-centric explainable AI-based systems. context-aware applications. In Proceedings of the 12th ACM international conference on Ubiquitous computing (pp. 13-22). ACM. [26] Lim, B. Y., & Dey, A. K. (2011a, September). Investigating intelligibility for REFERENCES uncertain context-aware applications. In Proceedings of the 13th international [1] Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., Kankanhalli, M. 2018. Trends conference on Ubiquitous computing (pp. 415-424). ACM. and Trajectories for Explainable, Accountable and Intelligible Systems: An [27] Lim, B. Y., & Dey, A. K. (2011b, August). Design of an intelligible mobile HCI Research Agenda. In Proceedings of the SIGCHI Conference on Human context-aware application. In Proceedings of the 13th international Factors in Computing Systems. CHI '18. conference on human computer interaction with mobile devices and [2] Anderson, H. (2015). Scientific Method. The Stanford Encyclopedia of services (pp. 157-166). ACM. Philosophy. https://plato.stanford.edu/entries/scientific-method/. Retrieved [28] Lim, B. Y. (2012). Improving understanding and trust with intelligibility in 10 September 2018. context-aware applications. PhD dissertation. CMU. [3] Assad, M., Carmichael, D. J., Kay, J., & Kummerfeld, B. (2007, May). [29] Lim, B. Y., & Dey, A. K. (2013, July). Evaluating Intelligibility Usage and PersonisAD: Distributed, active, scrutable model framework for context- Usefulness in a Context-Aware Application. In International Conference on aware services. In International Conference on Pervasive Computing (pp. 55- Human-Computer Interaction (pp. 92-101). Springer, Berlin, Heidelberg. 72). Springer, Berlin, Heidelberg. [30] Lipton, Z. C. (2016). The mythos of model interpretability. arXiv preprint arXiv:1606.03490. 6 Why these Explanations? ExSS '19, March 20, 2019, Los Angeles, CA, USA. Selecting Intelligibility Types for Explanation Goals [31] Miller, T. (2017). Explanation in artificial intelligence: insights from the social Annual Meeting of the Association for Computational Linguistics (Volume 1: sciences. arXiv preprint arXiv:1706.07269. Long Papers) (Vol. 1, pp. 856-865). [32] Nunes, I., & Jannach, D. (2017). A systematic review and taxonomy of [39] Roth-Berghofer, T. R. (2004, August). Explanations and case-based reasoning: explanations in decision support and recommender systems. User Modeling Foundational issues. In European Conference on Case-Based Reasoning (pp. and User-Adapted Interaction, 27(3-5), 393-444. 389-403). Springer, Berlin, Heidelberg. [33] Patel, V. L., Arocha, J. F., & Zhang, J. (2005). Thinking and reasoning in [40] Silveira, M.S., de Souza, C.S., and Barbosa, S.D.J. (2001). Semiotic engineering medicine. The Cambridge handbook of thinking and reasoning, 14, 727-750. contributions for designing online help systems. In Proceedings of the 19th [34] Peirce, C. S. (1903). Harvard lectures on pragmatism, Collected Papers v. 5. annual international conference on Computer documentation (SIGDOC '01) . [35] Popper, Karl (2002), Conjectures and Refutations: The Growth of Scientific ACM, New York, NY, USA, 31-38. Knowledge, London, UK: Routledge. [41] Vickers, John (2009). The Problem of Induction. The Stanford Encyclopedia of [36] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). Why should i trust Philosophy. https://plato.stanford.edu/entries/induction-problem/. Retrieved you?: Explaining the predictions of any classifier. In Proceedings of the 22nd 10 September 2018. ACM SIGKDD International Conference on Knowledge Discovery and Data [42] Wang, D., Yang, Q., Abdul, A., Lim, B.Y. 2019. Designing Theory-Driven User- Mining (pp. 1135-1144). ACM. Centric Explainable AI. In Proceedings of the SIGCHI Conference on Human [37] Ribeiro, M. T., Singh, S., & Guestrin, C. (2018a). Anchors: High-precision Factors in Computing Systems. CHI '19. model-agnostic explanations. In AAAI Conference on Artificial Intelligence. https://doi.org/10.1145/3290605.3300831 [38] Ribeiro, M. T., Singh, S., & Guestrin, C. (2018b). Semantically Equivalent [43] Zhang, Q. S., & Zhu, S. C. (2018). Visual interpretability for deep learning: a Adversarial Rules for Debugging NLP Models. In Proceedings of the 56th survey. Frontiers of Information Technology & Electronic Engineering , 19(1), 27-39. 7