We present a framework for defining the “right” level of explainability based on technical, legal and economic considerations. Our approach involves three logical steps: First, define the main contextual factors, such as who is the audience of the explanation, the operational context, the level of harm that the system could cause, and the legal/regulatory framework. This step will help characterize the operational and legal needs for explanation, and the corresponding social benefits. Second, examine the technical tools available, including post-hoc approaches (input perturbation, saliency maps...) and hybrid AI approaches. Third, as function of the first two steps, choose the right levels of global and local explanation outputs, taking into the account the costs involved. We identify seven kinds of costs and emphasize that explanations are socially useful only when total social benefits exceed costs.
This paper summarizes the conclusions of a longer paper [
Our approach involves three logical steps: First, define the main contextual factors, such as who is the audience of the explanation, the operational context, the level of harm that the system could cause, and the legal/regulatory framework. This step will help characterize the operational and legal needs for explanation, and the corresponding social benefits. Second, examine the technical tools available, 1 Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1. I3, Te´le´com Paris, CNRS, Institut Polytechnique de Paris, France – 2. LTCI, Te´le´com Paris, Institut Polytechnique de Paris, France – email: isabelle.bloch@telecom-paris.fr including post-hoc approaches (input perturbation, saliency maps...) and hybrid AI approaches. Third, as function of the first two steps, choose the right levels of global and local explanation outputs, taking into the account the costs involved.
The use of hybrid solutions, combining machine learning and symbolic AI, is a promising field of research for safety-critical applications, and applications such as medicine where important bodies of domain knowledge must be associated with algorithmic decisions. As technical solutions to explainability converge toward hybrid AI approaches, we can expect that the trade-off between explainability and performance will become less acute. Explainability will become part of performance. Also, as explainability becomes a requirement for safety certification, we can expect an alignment between operational/safety needs for explainability and ethical/human rights needs for explainability. Some of the solutions for operational explainability may serve both purposes. 2
Although several different definitions exist in the literature [
Transparency is a broader concept than explainability [
Our approach identifies three considerations that will help lead to the right level of explainability: the contextual factors (an input), the available technical solutions (an input), and the explainability choices regarding the form and detail of explanations (the outputs). 3.1
We have identified four kinds of contextual factors that will help
identify the various reasons why we need explanations and choose
the most appropriate form of explanation (output) as a function of
the technical possibilities and costs. The four contextual factors are:
Audience factors: Who is receiving the explanation? What is their
level of expertise? What are their time constraints? These will
profoundly impact the level of detail and timing of the
explanation [
Impact factors: What harms could the algorithm cause and how
might explanations help? These will determine the level of social
benefits associated with the explanation. Generally speaking, the
higher the impact of the algorithm, the higher the benefits flowing
from explanation [
Regulatory factors: What is the regulatory environment for the application? What fundamental rights are affected? These factors are examined in Section 5 and will help characterize the social benefits associated with an explanation in a given context.
Operational factors: To what extent is explanation an operational imperative? For safety certification? For user trust? These factors may help identify solutions that serve both operational and ethical/legal purposes. 3.2
Another input factor relates to the technical solutions available
for explanations. Post-hoc approaches such as LIME [
Modifying objective or predictor function;
Producing fuzzy rules, close to natural language;
Output approaches [
Genetic fuzzy logic.
The range of potential hybrid approaches, i.e. approaches that combine machine learning and symbolic or logic-based approaches, is almost unlimited. The examples above represent only a small selection. Most of the approaches, whether focused on inputs, outputs, or constraints within the model, can contribute to explainability, albeit in different ways. Explainability by design mostly aims at incorporating explainability in the predictor model. 3.3
The output of explanation will be what is actually shown to the relevant explanation audience, whether through global explanation of the algorithm’s operation, or through local explanation of a particular decision.
The output choices for global explanations will include the following:
Adoption of a “user’s manual” approach to present the functioning
of the algorithm as a whole [
The output choices for local explanations will include the following:
Counterfactual dashboards, with “what if” experimentation
available for end-users [
What information should be stored in logs, and for how long? 4
Much of the work on explainability in the 1990s, as well as the new industrial interest in explainability today, focus on explanations needed to satisfy users’ operational requirements. For example, the customer may require explanations as part of the safety validation and certification process for an AI system, or may ask that the system provide additional information to help the end user (for example, a radiologist) put the system’s decision into a clinical context.
These operational requirements for explainability may be required
to obtain certifications for safety-critical applications, since the
system could not go to market without those certifications. Customers
may also insist on explanations in order to make the system more
user-friendly and trusted by users. Knowing which factors cause
certain outcomes increases the system’s utility because the decisions
are accompanied by actionable insights, which can be much more
valuable than simply having highly-accurate but unexplained
predictions [
Explainability has an important role in algorithmic quality
control, both before the system goes to market and afterwards, because
it helps bring to light weaknesses in the algorithm such as bias that
would otherwise go unnoticed [
The quality of machine learning models is often judged by the
average accuracy rate when analyzing test data. This simple
measure of quality fails to reflect weaknesses affecting the algorithm’s
quality, particularly bias and failure to generalize. Explainability
solutions presented can assist in identifying areas of input data where
the performance of the algorithm is poor, and identify defects in the
learning data that lead to bad predictions. Traditional approaches to
software verification and validation (V&V) are ill-adapted to
neural networks [
The legal approaches to explanation are different for government decisions and for private sector decisions. The obligation for governments to give explanations has constitutional underpinnings, for example the right to due process under the United States Constitution, and the right to challenge administrative decisions under European human rights instruments. These rights require that individuals and courts be able to understand the reasons for algorithmic decisions, replicate the decisions to test for errors, and evaluate the proportionality of systems in light of other affected human rights such as the right to privacy. In the United States, the Houston Teachers case2 illustrates how explainability is linked to the constitutional guarantee of due process. In Europe, the Hague District Court decision on the SyLI algorithm3 shows how explainability is closely linked to the European constitutional principle of proportionality. France has enacted a law on government-operated algorithms4, which includes particularly stringent explainability requirements: disclosure of the degree and manner in which the algorithmic processing contributed to the decision; the data used for the processing and their source; the parameters used and their weights in the individual processing; and the operations effected by the processing.
For private entities, a duty of explanation generally arises when the entity becomes subject to a heightened duty of fairness or loyalty, which can happen when the entity occupies a dominant position under antitrust law, or when it occupies functions that create a situation of trust or dependency vis a` vis users. A number of specific laws impose algorithmic explanations in the private sector. One of the most recent is Europe’s Platform to Business Regulation (EU) 2018/1150, which imposes a duty of explanation on online intermediaries and search engines with regard to ranking algorithms. The language in the regulation shows the difficult balance between competing principles: providing complete information, protecting trade secrets, avoiding giving information that would permit bad faith manipulation of ranking algorithms by third parties, and making explanations easily understandable and useful for users. Among other things, online intermediaries and search engines must provide a “reasoned description” of the “main parameters” affecting ranking on the platform, including the “general criteria, processes, specific signals incorporated into algorithms or other adjustment or demotion mechanisms 2 Local 2415 v. Houston Independent School District, 251 F. Supp. 3d 1168 (S.D. Tex. 2017). 3 NJCM v. the Netherlands, District Court of The Hague, Case n. C-09550982-HA ZA 18-388, February 5, 2020. 4 French Code of Relations between the Public and the Administration, articles L. 311-3-1 et seq. used in connection with the ranking.”5 These requirements are more detailed than those in Europe’s General Data Protection Regulation EU 2016/679 (GDPR), which requires only “meaningful information about the logic involved.”6 In the United States, banks already have an obligation to provide the principal reasons for any denial of a loan.7 A proposed bill in the United States called the Algorithmic Accountability Act would impose explainability obligations on certain high-impact algorithms, including an obligation to provide “detailed description of the automated decision system, its design, its training, data, and its purpose.”8 6
Laws and regulations generally impose explanations when doing so
is socially beneficial, that is, when the collective benefits associated
with providing explanations exceed the costs. When considering
algorithmic explainability, where the law has not yet determined
exactly what form of explainability is required and in which context,
the costs and benefits of explanations will help fill the gaps and define
the right level of explanation. The cost-benefit analysis will help
determine when and how explanations should be provided, permitting
various trade-offs to be highlighted and managed. For explanations to
be socially useful, benefits should always exceed the costs. The
benefits of explanations are closely linked to the level of impact of the
algorithm on individual and collective rights [
Explanations generate many kinds of costs, some of which are not obvious. We have identified seven categories of costs:
Design and integration costs, which may be high because
explanation requirements will vary among different applications, contexts
and geographies, meaning that a one-size-fits-all explanation
solution will rarely be sufficient [
For high-impact algorithmic decisions, these costs will often be outweighed by the benefits of explanations. But the costs should nevertheless be considered in each case to ensure that the form and level 5 Regulation 2018/1150, recital 24. 6 Regulation 2016/679, article 13(2)(f). 7 12 CFR Part 1002.9. 8 Proposed Algorithmic Accountability Act, H.R. 2231, introduced April 10, 2019. of detail of mandated explanations is adapted to the situation. The net social benefit (total benefits less total costs) should remain positive. 7
Regulation of AI explainability remains largely unexplored territory, the most ambitious efforts to date being the French law on the explainability of government algorithms and the EU regulation on Platform to Business relations. However, even in those instances, the law leaves many aspects of explainability open to interpretation. The form of explanation and the level of detail will be driven by the four categories of contextual factors described in this paper: audience factors, impact factors, regulatory factors, and operational factors. The level of detail of explanations – global or local – would follow a sliding scale depending on the context, and the costs and benefits at stake. One of the biggest costs of local explanations will relate to storage of individual decision logs. The kind of information stored in the logs, and the duration of storage, will be key questions to address when determining the right level of explainability. Hybrid solutions attempt to create explainability by design, mostly by incorporating explainability in the predictor model. While generally addressing operational needs, these hybrid approaches may also serve ethical and legal explainability needs. Our three-step method involving contextual factors, technical solutions, and explainability outputs will help lead to the “right” level of explanation in a given situation.
Future work aims at instantiating the proposed three steps to realistic and concrete problems, to give insight in the feasibility and value of the method to provide the right level of explanation.