We characterize three notions of explainable AI that cut across research elds: opaque systems that o er no insight into its algorithmic mechanisms; interpretable systems where users can mathematically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing di erences in how work on explainable AI is positioned in various elds. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as nal step of the generative process.
If you were held accountable for the decision of a machine in contexts that have
nancial, safety, security, or personal rami cations to an individual, would you
blindly trust its decision? How can we hold accountable Arti cial Intelligence
(AI) systems that make decisions on possibly unethical grounds, e.g. when they
predict a person's weight and health by their social media images [
To overcome this dangerous practice, it is prudent for an AI to provide not only an output, but also a human understandable explanation that expresses the rationale of the machine. Analysts can turn to such explanations to evaluate if a decision is reached by rational arguments and does not incorporate reasoning steps con icting with ethical or legal norms.
But what constitutes an explanation? The Oxford English dictionary has
no entry for the term `explainable', but has one for explanation: A statement
or account that makes something clear; a reason or justi cation given for an
action or belief. Do present systems that claim to make `explainable' decisions
really provide explanations? Those who argue yes may point to Machine Learning
(ML) algorithms that produce rules about data features to establish a classi
cation decision, such as those learned by decision trees [
ACL As stated by Lipton, terms like in1:4 CONGIPSSCI terpretability are used in research pa1:2 ICCV/ECCV pers, despite the lack of a clear and ecyn 1 qwuidaenltyifyshatrheids doebsneritviaotnio.nI,n woredesrutgoeruq 0:8 gest a corpus-based analysis of releferm 0:6 vant terms across research communiT 0:4 ties which strongly rely on ML-driven 0:2 methodologies. The goal of this anal0 2007 2008 2009 2010 201Y1ea20r12 2013 2014 2015 2016 eyvsaisncise toof geaxinplaininsaigbhiltistyinatocrothsse rAeIl-related research communities and to Fig. 1: Normalized corpus frequency of detect how each elds de nes notions \explain" or \explanation". of explainability. We carried out an experiment over corpora of papers from the computer vision, NLP, connectionist, and symbolic reasoning communities. We base our analysis on corpus statistics compiled from the proceedings of conferences where researchers employ, inter alia, ML techniques to approach their research objectives: the Annual Meeting of the Association for Computational Linguistics (ACL), the Annual Conference on Neural Information Processing Systems (NIPS), the International/European Conference on Computer Vision (ICCV/ECCV), and the Annual Conference of the Cognitive Science Society (COGSCI). The corpora include proceedings from 2007 to 2016. This allows us to observe trends regarding the use of words related to various concepts and the scope these concepts take. We perform a shallow linguistic search for what we henceforth call \explanation terms". We apply simple substring matches such as \explain" or \explanation" for explainability, \interpret" for interpretability and \compreh" for comprehensibility. \Explanation terms" serve as an approximation to aspects of explainability. Frequencies are normalized, making them comparable between years and conferences.
The normalized frequencies for explanation terms are plotted in Figure 1. We omit frequency plots for interpretation and comprehension terms because they exhibit a similar pattern. The frequency level of explainability concepts for COGSCI is signi cantly above those of the other corpora. This could be due to the fact that Cognitive Science explicitly aims to explain the mind and its processes, in many cases leading to qualitatively di erent research questions, corresponding terminology, and actively used vocabularies. The NIPS corpus hints at an emphasis of explainability in 2008 and a slight increase in interest in this concept in 2016 in the connectionist community. To better understand how consistent topics and ideas around explainability are across elds, we also analyze the context of its mentions. Word clouds shown in Figure 2 are a simple method to gain an intuition about the composition of a concept and its semantic contents by highlighting the important words related to it. Important words are de ned as words that appear within a 20 words window of a mention of an explanation term with a frequency highly above average4.
All communities focus on the explainability of a model but there is a difference between the nature of models in Cognitive Science and the other elds. The COGSCI corpus mentions a participant, a task and an e ect whereas the other communities focus on a model and what constitutes data in their elds. It is not surprising that the neural modeling and NLP communities show a large overlap in their usage of explainability since there is an overlap in the research communities as well. We note further di erences across the three ML communities compared to COGSCI. In the ACL corpus, explainability is often paired with words like features, examples, and words, which could suggest an emphasis on using examples to demonstrate the decision making of NLP decisions and the relevance of particular features. In the NIPS corpus, explainability is more closely tied to methods, algorithms, and results suggesting a desire to establish explanations about how neural systems translate inputs to outputs. The ICCV/ECCV falls between the ACL and NIPS corpus in the sense that it pairs explainability with data (images) and features (objects) like ACL, but may also tie the notion to how algorithms use (using) images to generate outputs.
The corpus analysis establishes some di erences in how various AI communities approach the concept of explainability. In particular, we note that the term 4 Word clouds are generated with the word-cloud package (http://amueller.github.
io/word_cloud/index.html). is sometimes used to help probe the mechanisms of ML systems (e.g. we seek an interpretation of how the system works), and other times to relate explanations to particular inputs and examples (e.g. we want to comprehend how an input was mapped to an output). We use these observations to develop the following notions, also illustrated in Figure 3:
Opaque systems. A system where the mechanisms mapping inputs to outputs are invisible to the user. It can be seen as an oracle that makes predictions over an input, without indicating how and why predictions are made. Opaque systems emerge, for instance, when closed-source AI is licensed by an organization, where the licensor does not want to reveal the workings of its proprietary AI. Similarly, systems relying on genuine \black box" approaches, for which inspection of the algorithm or implementation does not give insight into the system's actual reasoning from inputs to corresponding outputs, are classi ed as opaque.
Interpretable systems. A system where a user cannot only see, but also study and understand how inputs are mathematically mapped to outputs. This implies model transparency, and requires a level of understanding of the technical details of the mapping. A regression model can be interpreted by comparing covariate weights to realize the relative importance of each feature to the mapping. SVMs and other linear classi ers are interpretable insofar as data classes are de ned by their location relative to decision boundaries. But the action of deep neural networks, where input features may be automatically learned and transformed through non-linearities, is unlikely to be interpretable by most users.
Comprehensible systems. A comprehensible system emits symbols along
with its output (echoing Michie's strong and ultra-strong machine learning [
The arrows in Figure 3 suggest that comprehensible and interpretable systems
each are improvements over opaque systems. The notions of comprehension and
interpretation are separate: while interpretation requires transparency in the
underlying mechanisms of a system, a comprehensible one can be opaque while
emitting symbols a user can reason over. Regression models, support vector
machines, decision trees, ANOVAs, and data clustering (assuming a kernel that
is itself interpretable) are common examples of interpretable models. High
dimensional data visualizations like t-SNE [
It is important that research in both interpretable and comprehensible systems continue forward. This is because, depending on the user's background and her purpose of employing an AI model, one type is preferable to another. As a real-life example of this, most people think of a doctor as a kind of black box that transforms symptoms and test results into a diagnosis. Without providing information about the way medical tests and evaluations work, doctors deliver a diagnosis to a patient by explaining high-level indicators revealed in the tests (i.e. system symbols). Thus, when facing a patient, the physician should be like a comprehensible model. When interacting with other doctors and medical sta , however, the doctor may be like an interpretable model: She can sketch a technical line of connecting patient symptoms and test results to a particular diagnosis. Other doctors and sta can interpret a diagnosis in the same way that an analyst can interpret an ML model, ensuring that the conclusions drawn are supported by reasonable evaluation functions and weight values for the evidence presented.
Explainable system traits. Often discussed alongside explainable AI are
the external traits such systems should exhibit. These traits are seen as so
important that some authors argue an AI system is not `explainable' if it does not
support them [
We also note that \completeness" is not an explicit trait, and might not even be desirable as such. Continuing with the doctor example from above, it may be desirable for a system to present a simpli ed (in the sense of incomplete, as opposed to abstracted) `explanation' similar to a doctor using a patient's incomplete and possibly not entirely accurate preconceptions in explaining a complex diagnosis, or even sparing the patient especially worrisome details which might not be relevant for the subsequent treatment.
Interpretable and comprehensible models encompass much of the present work in explainable AI. Yet we argue that both approaches are lacking in their ability to formulate, for the user, a line of reasoning that explains the decision making process of a model using human-understandable features of the input data. Reasoning is a critical step in formulating an explanation about why or how some event has occurred (see, e.g., Figure 5). Leaving explanation generation to human analysts can be dangerous since, depending on their background knowledge about the data and its domain, di erent explanations about why a model makes a decision may be deduced. Interpretive and comprehensible models thus enable explanations of decisions, but do not yield explanations themselves.
E orts in neural-symbolic integration [