=Paper=
{{Paper
|id=Vol-2659/beaudoin
|storemode=property
|title=Identifying the "right" level of explanation in
a given situation
|pdfUrl=https://ceur-ws.org/Vol-2659/beaudoin.pdf
|volume=Vol-2659
|authors=Valérie Beaudoin,Isabelle Bloch,David Bounie,Stéphan Clémençon,Florence D'alché-Buc,James Eagan,Winston Maxwell,Pavlo Mozharovskyi,Jayneel Parekh
|dblpUrl=https://dblp.org/rec/conf/ecai/BeaudouinBBCDEM20
}}
==Identifying the "right" level of explanation in
a given situation==
https://ceur-ws.org/Vol-2659/beaudoin.pdf
IDENTIFYING THE “RIGHT” LEVEL OF
EXPLANATION IN A GIVEN SITUATION
Valérie Beaudouin1 and Isabelle Bloch2 and David Bounie1 and Stéphan Clémençon2 and
Florence d’Alché-Buc2 and James Eagan2 and Winston Maxwell1 and
Pavlo Mozharovskyi2 and Jayneel Parekh2 1
Abstract. We present a framework for defining the “right” level of including post-hoc approaches (input perturbation, saliency maps...)
explainability based on technical, legal and economic considerations. and hybrid AI approaches. Third, as function of the first two steps,
Our approach involves three logical steps: First, define the main con- choose the right levels of global and local explanation outputs, taking
textual factors, such as who is the audience of the explanation, the into the account the costs involved.
operational context, the level of harm that the system could cause, The use of hybrid solutions, combining machine learning and sym-
and the legal/regulatory framework. This step will help characterize bolic AI, is a promising field of research for safety-critical applica-
the operational and legal needs for explanation, and the correspond- tions, and applications such as medicine where important bodies of
ing social benefits. Second, examine the technical tools available, domain knowledge must be associated with algorithmic decisions.
including post-hoc approaches (input perturbation, saliency maps...) As technical solutions to explainability converge toward hybrid AI
and hybrid AI approaches. Third, as function of the first two steps, approaches, we can expect that the trade-off between explainability
choose the right levels of global and local explanation outputs, taking and performance will become less acute. Explainability will become
into the account the costs involved. We identify seven kinds of costs part of performance. Also, as explainability becomes a requirement
and emphasize that explanations are socially useful only when total for safety certification, we can expect an alignment between opera-
social benefits exceed costs. tional/safety needs for explainability and ethical/human rights needs
for explainability. Some of the solutions for operational explainabil-
ity may serve both purposes.
1 INTRODUCTION
This paper summarizes the conclusions of a longer paper [1] on 2 DEFINITIONS
context-specific explanations using a multidisciplinary approach. Ex- Although several different definitions exist in the literature [1], we
plainability is both an operational and ethical requirement. The op- have treated explainability and interpretability as synonyms [16], fo-
erational needs for explainability are driven by the need to increase cusing instead on the key difference between “global” and “local”
robustness, particularly for safety-critical applications, as well as en- explainability/interpretability. Global explainability means the abil-
hance acceptance by system users. The ethical needs for explainabil- ity to explain the functioning of the algorithm in its entirety, whereas
ity address harms to fundamental rights and other societal interests local explainability means the ability to explain a particular algorith-
which may be insufficiently addressed by the purely operational re- mic decision [7]. Local explainability is also known as “post hoc”
quirements. Existing works on explainable AI focus on the computer explainability.
science angle [18], or on the legal and policy angle [20]. The origi- Transparency is a broader concept than explainability [6], because
nality of this paper is to integrate technical, legal and economic ap- transparency includes the idea of providing access to raw informa-
proaches into a single methodology for reaching the optimal level of tion whether or not the information is understandable. By contrast,
explainability. The technical dimension helps us understand what ex- explainability implies a transformation of raw information in order
planations are possible and what the trade-offs are between explain- to make it understandable by humans. Thus explainability is a value-
ability and algorithmic performance. However explanations are nec- added component of transparency. Transparency and explainability
essarily context-dependent, and context depends on the regulatory do not exist for their own sake. Instead, they are enablers of other
environment and a cost-benefit analysis, which we discuss below. functions such as traceability and auditability, which are critical in-
Our approach involves three logical steps: First, define the main puts to accountability. In a sense, accountability is the nirvana of al-
contextual factors, such as who is the audience of the explanation, gorithmic governance [15] into which other concepts, including ex-
the operational context, the level of harm that the system could cause, plainability, feed.
and the legal/regulatory framework. This step will help characterize
the operational and legal needs for explanation, and the correspond-
3 THREE FACTORS DETERMINING THE
ing social benefits. Second, examine the technical tools available,
“RIGHT” LEVEL OF EXPLANATION
1 Copyright c 2020 for this paper by its authors. Use permitted under Our approach identifies three considerations that will help lead to
Creative Commons License Attribution 4.0 International (CC BY 4.0).
1. I3, Télécom Paris, CNRS, Institut Polytechnique de Paris, France –
the right level of explainability: the contextual factors (an input),
2. LTCI, Télécom Paris, Institut Polytechnique de Paris, France – email: the available technical solutions (an input), and the explainability
isabelle.bloch@telecom-paris.fr choices regarding the form and detail of explanations (the outputs).
3.1 Contextual factors • Whether to provide access to source code, taking into account
trade secret protection and the sometimes limited utility of source
We have identified four kinds of contextual factors that will help code to the relevant explanation audience [10, 20];
identify the various reasons why we need explanations and choose • Information on training data, including potentially providing a
the most appropriate form of explanation (output) as a function of copy of the training data [10, 13, 17];
the technical possibilities and costs. The four contextual factors are: • Information on the learning algorithm, including its objective
function;
• Audience factors: Who is receiving the explanation? What is their
• Information on known biases and other inherent weaknesses of the
level of expertise? What are their time constraints? These will
algorithm; identifying use restrictions and warnings.
profoundly impact the level of detail and timing of the explana-
tion [5, 7].
The output choices for local explanations will include the follow-
• Impact factors: What harms could the algorithm cause and how
ing:
might explanations help? These will determine the level of social
benefits associated with the explanation. Generally speaking, the
• Counterfactual dashboards, with “what if” experimentation avail-
higher the impact of the algorithm, the higher the benefits flowing
able for end-users [20, 24];
from explanation [8].
• Saliency maps to show the main factors contributing to decision;
• Regulatory factors: What is the regulatory environment for the ap-
• Defining the level of detail, including how many factors and rele-
plication? What fundamental rights are affected? These factors are
vant weights to present to end-users;
examined in Section 5 and will help characterize the social bene-
• Layered explanation tools, permitting a user to access increasing
fits associated with an explanation in a given context.
levels of complexity;
• Operational factors: To what extent is explanation an operational
• Access to individual decision logs [11, 26];
imperative? For safety certification? For user trust? These factors
• What information should be stored in logs, and for how long?
may help identify solutions that serve both operational and ethi-
cal/legal purposes.
4 EXPLAINABILITY AS AN OPERATIONAL
3.2 Technical solutions REQUIREMENT
Another input factor relates to the technical solutions available Much of the work on explainability in the 1990s, as well as the
for explanations. Post-hoc approaches such as LIME [18], Kernal- new industrial interest in explainability today, focus on explanations
SHAP [14] and saliency maps [21] generally strive to approximate needed to satisfy users’ operational requirements. For example, the
the functioning of a black-box model by using a separate explanation customer may require explanations as part of the safety validation
model. Hybrid approaches tend to incorporate the need for explana- and certification process for an AI system, or may ask that the sys-
tion into the model itself. These approaches include: tem provide additional information to help the end user (for example,
a radiologist) put the system’s decision into a clinical context.
• Modifying objective or predictor function; These operational requirements for explainability may be required
• Producing fuzzy rules, close to natural language; to obtain certifications for safety-critical applications, since the sys-
• Output approaches [22]; tem could not go to market without those certifications. Customers
• Input approaches, which pre-process the inputs to the machine may also insist on explanations in order to make the system more
learning model, making the inputs more meaningful and/or bet- user-friendly and trusted by users. Knowing which factors cause cer-
ter structured [1]; tain outcomes increases the system’s utility because the decisions
• Genetic fuzzy logic. are accompanied by actionable insights, which can be much more
valuable than simply having highly-accurate but unexplained pre-
The range of potential hybrid approaches, i.e. approaches that com- dictions [25]. Understanding causality can also enhance quality by
bine machine learning and symbolic or logic-based approaches, is making models more robust to shifting input domains. Customers
almost unlimited. The examples above represent only a small selec- increasingly consider explainability as a quality feature for the AI
tion. Most of the approaches, whether focused on inputs, outputs, or system. These operational requirements are distinct from regulatory
constraints within the model, can contribute to explainability, albeit demands for explainability, which we examine in Section 5, but may
in different ways. Explainability by design mostly aims at incorpo- nevertheless lead to a convergence in the tools used to meet the vari-
rating explainability in the predictor model. ous requirements.
Explainability has an important role in algorithmic quality con-
trol, both before the system goes to market and afterwards, because
3.3 Explanation output choices it helps bring to light weaknesses in the algorithm such as bias that
The output of explanation will be what is actually shown to the rel- would otherwise go unnoticed [9]. Explainability contributes to “to-
evant explanation audience, whether through global explanation of tal product lifecycle” [23] or “safety lifecycle” [12] approaches to
the algorithm’s operation, or through local explanation of a particu- algorithmic quality and safety.
lar decision. The quality of machine learning models is often judged by the
The output choices for global explanations will include the fol- average accuracy rate when analyzing test data. This simple mea-
lowing: sure of quality fails to reflect weaknesses affecting the algorithm’s
quality, particularly bias and failure to generalize. Explainability so-
• Adoption of a “user’s manual” approach to present the functioning lutions presented can assist in identifying areas of input data where
of the algorithm as a whole [10]; the performance of the algorithm is poor, and identify defects in the
• The level of detail to include in the user’s manual; learning data that lead to bad predictions. Traditional approaches to
software verification and validation (V&V) are ill-adapted to neu- used in connection with the ranking.”5 These requirements are more
ral networks [3, 17, 23]. The challenges relate to neural networks’ detailed than those in Europe’s General Data Protection Regulation
non-determinism, which makes it hard to demonstrate the absence EU 2016/679 (GDPR), which requires only “meaningful informa-
of unintended functionality, and to the adaptive nature of machine- tion about the logic involved.”6 In the United States, banks already
learning algorithms [3, 23]. Specifying a set of requirements that have an obligation to provide the principal reasons for any denial of a
comprehensively describe the behavior of a neural network is con- loan.7 A proposed bill in the United States called the Algorithmic Ac-
sidered the most difficult challenge with regard to traditional V&V countability Act would impose explainability obligations on certain
and certification approaches [2, 3]. The absence of complete require- high-impact algorithms, including an obligation to provide “detailed
ments poses a problem because one of the objectives of V&V is to description of the automated decision system, its design, its training,
compare the behavior of the software to a document that describes data, and its purpose.”8
precisely and comprehensively the system’s intended behavior [17].
For neural networks, there may remain a degree of uncertainty about
just what will be the output for a given input.
6 THE BENEFITS AND COSTS OF
EXPLANATIONS
Laws and regulations generally impose explanations when doing so
5 EXPLAINABILITY AS A LEGAL is socially beneficial, that is, when the collective benefits associated
REQUIREMENT with providing explanations exceed the costs. When considering al-
The legal approaches to explanation are different for government de- gorithmic explainability, where the law has not yet determined ex-
cisions and for private sector decisions. The obligation for govern- actly what form of explainability is required and in which context,
ments to give explanations has constitutional underpinnings, for ex- the costs and benefits of explanations will help fill the gaps and define
ample the right to due process under the United States Constitution, the right level of explanation. The cost-benefit analysis will help de-
and the right to challenge administrative decisions under European termine when and how explanations should be provided, permitting
human rights instruments. These rights require that individuals and various trade-offs to be highlighted and managed. For explanations to
courts be able to understand the reasons for algorithmic decisions, be socially useful, benefits should always exceed the costs. The ben-
replicate the decisions to test for errors, and evaluate the proportion- efits of explanations are closely linked to the level of impact of the
ality of systems in light of other affected human rights such as the algorithm on individual and collective rights [5, 8]. For algorithms
right to privacy. In the United States, the Houston Teachers case2 with low impact, such as a music recommendation algorithms, the
illustrates how explainability is linked to the constitutional guaran- benefits of explanation will be low. For a high-impact algorithm such
tee of due process. In Europe, the Hague District Court decision on as the image recognition algorithm of an autonomous vehicle, the
the SyLI algorithm3 shows how explainability is closely linked to benefits of explanation, for example in finding the cause of a crash,
the European constitutional principle of proportionality. France has will be high.
enacted a law on government-operated algorithms4 , which includes Explanations generate many kinds of costs, some of which are not
particularly stringent explainability requirements: disclosure of the obvious. We have identified seven categories of costs:
degree and manner in which the algorithmic processing contributed • Design and integration costs, which may be high because explana-
to the decision; the data used for the processing and their source; the tion requirements will vary among different applications, contexts
parameters used and their weights in the individual processing; and and geographies, meaning that a one-size-fits-all explanation so-
the operations effected by the processing. lution will rarely be sufficient [9];
For private entities, a duty of explanation generally arises when • Sacrificing prediction accuracy for the sake of explainability
the entity becomes subject to a heightened duty of fairness or loyalty, can result in lower performance, thereby generating opportunity
which can happen when the entity occupies a dominant position un- costs [5];
der antitrust law, or when it occupies functions that create a situation • The creation and storage of decision logs create operational costs
of trust or dependency vis à vis users. A number of specific laws im- but also tensions with data privacy principles which generally re-
pose algorithmic explanations in the private sector. One of the most quire destruction of logs as soon as possible [11, 26];
recent is Europe’s Platform to Business Regulation (EU) 2018/1150, • Forced disclosure of source code or other algorithmic details may
which imposes a duty of explanation on online intermediaries and interfere with constitutionally-protected trade secrets [4];
search engines with regard to ranking algorithms. The language in • Detailed explanations on the functioning of an algorithm can fa-
the regulation shows the difficult balance between competing princi- cilitate gaming of the system and result in decreased security;
ples: providing complete information, protecting trade secrets, avoid- • Explanations create implicit rules and precedents, which the de-
ing giving information that would permit bad faith manipulation of cision maker will have to take into account in the future, thereby
ranking algorithms by third parties, and making explanations eas- limiting her decisional flexibility in the future [19];
ily understandable and useful for users. Among other things, online • Mandating explainability can increase time to market, thereby
intermediaries and search engines must provide a “reasoned descrip- slowing innovation [9].
tion” of the “main parameters” affecting ranking on the platform,
including the “general criteria, processes, specific signals incorpo- For high-impact algorithmic decisions, these costs will often be
rated into algorithms or other adjustment or demotion mechanisms outweighed by the benefits of explanations. But the costs should nev-
2 Local 2415 v. Houston Independent School District, 251 F. Supp. 3d 1168
ertheless be considered in each case to ensure that the form and level
5 Regulation 2018/1150, recital 24.
(S.D. Tex. 2017).
3 NJCM v. the Netherlands, District Court of The Hague, Case n. C-09- 6 Regulation 2016/679, article 13(2)(f).
550982-HA ZA 18-388, February 5, 2020. 7 12 CFR Part 1002.9.
4 French Code of Relations between the Public and the Administration, arti- 8 Proposed Algorithmic Accountability Act, H.R. 2231, introduced April 10,
cles L. 311-3-1 et seq. 2019.
of detail of mandated explanations is adapted to the situation. The net [10] IEEE, ‘Ethically aligned design: A vision for prioritizing human well-
social benefit (total benefits less total costs) should remain positive. being with autonomous and intelligent systems’, IEEE Global Initiative
on Ethics of Autonomous and Intelligent Systems, (2019).
[11] Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg,
David G Robinson, and Harlan Yu, ‘Accountable algorithms’, U. Pa. L.
7 CONCLUSION: CONTEXT-SPECIFIC AI Rev., 165, 633, (2016).
EXPLANATIONS BY DESIGN [12] Zeshan Kurd and Tim Kelly, ‘Safety lifecycle for developing safety crit-
ical artificial neural networks’, in Computer Safety, Reliability, and Se-
Regulation of AI explainability remains largely unexplored territory, curity, eds., Stuart Anderson, Massimo Felici, and Bev Littlewood, pp.
the most ambitious efforts to date being the French law on the ex- 77–91, Berlin, Heidelberg, (2003). Springer Berlin Heidelberg.
plainability of government algorithms and the EU regulation on Plat- [13] David Lehr and Paul Ohm, ‘Playing with the data: what legal scholars
should learn about machine learning’, UCDL Rev., 51, 653, (2017).
form to Business relations. However, even in those instances, the [14] Scott M Lundberg and Su-In Lee, ‘A unified approach to interpreting
law leaves many aspects of explainability open to interpretation. The model predictions’, in Advances in Neural Information Processing Sys-
form of explanation and the level of detail will be driven by the four tems, pp. 4765–4774, (2017).
categories of contextual factors described in this paper: audience fac- [15] OECD, Artificial Intelligence in Society, 2019.
tors, impact factors, regulatory factors, and operational factors. The [16] OECD, Recommendation of the Council on Artificial Intelligence,
2019.
level of detail of explanations – global or local – would follow a [17] Gerald E Peterson, ‘Foundation for neural network verification and val-
sliding scale depending on the context, and the costs and benefits at idation’, in Science of Artificial Neural Networks II, volume 1966, pp.
stake. One of the biggest costs of local explanations will relate to 196–207. International Society for Optics and Photonics, (1993).
storage of individual decision logs. The kind of information stored in [18] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin, ‘Why should
I trust you?: Explaining the predictions of any classifier’, in 22nd ACM
the logs, and the duration of storage, will be key questions to address SIGKDD International Conference on Knowledge Discovery and Data
when determining the right level of explainability. Hybrid solutions Mining, pp. 1135–1144, (2016).
attempt to create explainability by design, mostly by incorporating [19] Frederick Schauer, ‘Giving reasons’, Stanford Law Review, 633–659,
explainability in the predictor model. While generally addressing op- (1995).
erational needs, these hybrid approaches may also serve ethical and [20] Andrew Selbst and Solon Barocas, ‘The intuitive appeal of explainable
machines’, SSRN Electronic Journal, 87, (01 2018).
legal explainability needs. Our three-step method involving contex- [21] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, ‘Deep in-
tual factors, technical solutions, and explainability outputs will help side convolutional networks: Visualising image classification models
lead to the “right” level of explanation in a given situation. and saliency maps’, arXiv preprint arXiv:1312.6034, (2013).
Future work aims at instantiating the proposed three steps to re- [22] Philip S. Thomas, Bruno Castro da Silva, Andrew G. Barto, Stephen
Giguere, Yuriy Brun, and Emma Brunskill, ‘Preventing undesirable be-
alistic and concrete problems, to give insight in the feasibility and havior of intelligent machines’, Science, 366(6468), 999–1004, (2019).
value of the method to provide the right level of explanation. [23] US Food and Drug Administration, ‘Proposed regulatory framework for
modifications to artificial intelligence/machine learning (AI/ML)-based
software as a medical device’, Technical report, (2019).
REFERENCES [24] Sandra Wachter, Brent Mittelstadt, and Chris Russell, ‘Counterfactual
explanations without opening the black box: Automated decisions and
[1] Valérie Beaudoin, Isabelle Bloch, David Bounie, Stéphan Clémencon, the gpdr’, Harv. JL & Tech., 31, 841, (2017).
Florence d’Aché Buc, James Eagan, Maxwell Winston, Pavlo [25] Max Welling, ‘Are ML and statistics complementary?’, in IMS-ISBA
Mozharovskyi, and Jayneel Parekh, ‘Flexible and context-specific AI Meeting on ‘Data Science in the Next 50 Years, (2015).
explainability: a multidisciplinary approach’, Technical report, ArXiv, [26] Alan FT Winfield and Marina Jirotka, ‘The case for an ethical black
(2020). box’, in Annual Conference Towards Autonomous Robotic Systems, pp.
[2] Siddhartha Bhattacharyya, Darren Cofer, D Musliner, Joseph Mueller, 262–273. Springer, (2017).
and Eric Engstrom, ‘Certification considerations for adaptive systems’,
in 2015 IEEE International Conference on Unmanned Aircraft Systems
(ICUAS), pp. 270–279, (2015).
[3] Markus Borg, Cristofer Englund, Krzysztof Wnuk, Boris Duran,
Christoffer Levandowski, Shenjian Gao, Yanwen Tan, Henrik Kaijser,
Henrik Lönn, and Jonas Törnqvist, ‘Safely entering the deep: A review
of verification and validation for machine learning and a challenge elic-
itation in the automotive industry’, Journal of Automotive Software En-
gineering, 1(1), 1–19, (2019).
[4] Jenna Burrell, ‘How the machine ‘thinks’: Understanding opac-
ity in machine learning algorithms’, Big Data & Society, 3(1),
2053951715622512, (2016).
[5] Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam
Gershman, David O’Brien, Stuart Schieber, James Waldo, David Wein-
berger, and Alexandra Wood, ‘Accountability of ai under the law: The
role of explanation’, arXiv preprint arXiv:1711.01134, (2017).
[6] European Commission, ‘Communication from the Commission to the
European Parliament, the Council, the European Economic and Social
Committee and the Committee of the Regions - Building trust in hu-
man centric artificial intelligence (com(2019)168)’, Technical report,
(2019).
[7] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini,
Fosca Giannotti, and Dino Pedreschi, ‘A survey of methods for explain-
ing black box models’, ACM Computing Surveys (CSUR), 51(5), 93,
(2018).
[8] AI HLEG, ‘High-level expert group on artificial intelligence’, Ethics
Guidelines for Trustworthy AI, (2019).
[9] ICO, ‘Project ExplAIn interim report’, Technical report, Information
Commissioner’s Office, (2019).