The assessment of AI-based systems trustworthiness is a challenging process given the complexity of the subject which involves qualitative and quantifiable concepts, a wide heterogeneity and granularity of attributes, and in some cases even the non-commensurability of the latter. Evaluating trustworthiness of AI-enabled systems is in particular decisive in safety-critical domains where AIs are expected to mostly operate autonomously. To overcome these issues, the Confiance.ai program [ 1] proposes an innovative solution based upon a multi-criteria decision analysis. The approach encompasses several phases: structuring trustworthiness as a set of well-defined attributes, the exploration of attributes to determine related performance metrics (or indicators), the selection of assessment methods or control points, and structuring a multi-criteria aggregation method to estimate a global evaluation of trust. The approach is illustrated by applying some performance metrics to a data-driven AI context whereas the focus on aggregation methods is left as a near-term perspective of Confiance.ai milestones.
from the early stages of development, the deployment of an Artificial Intelligence (AI) component within a safetycritical systems such as in avionics, mobility, healthcare and defense becomes risky.
Trust is the willingness of one party to perform certain
actions that are important to stakeholders (AI scientist,
safety engineer, certification auditor, end-user, etc.)
regardless of the other partys ability to monitor or control
[
SafeAI 2023: The AAAI’s Workshop on Artificial Intelligence Safety Trustworthiness is a complex concept which can be bro* Corresponding author: J. MATTIOLI ken down into diferent attributes. In 1977, a FAA (Fed†$Thjuelsieetatue.tmhoartstiocolin@trtihbaulteesdgreoquupa.lcloym. (J. MATTIOLI); eral Aviation Administration) panel dedicated to how to henri.sohier@itr-systemx.fr (H. SOHIER); agnes.delaborde@lne.fr certify aircraft as airworthy, explicitly linked the notion (A. DELABORDE); gabriel.pedroza@cea.fr (G. PEDROZA); of trustworthiness to accounting. Then, security and dekahina.amokrane-ferka@irt-systemx.fr (K. AMOKRANE-FERKA); pendability became key system attributes [6] to assess afef.awadid@irt-systemx.fr (A. AWADID); the trustworthiness of a computer-based system: AviziesZoaukhaariiae.lC.kHhaIHlfaAoNuIi@@cveaale.for.c(Zom.C(HS.IHKAHNAIL)F;AOUI) nis et al. [7] used dependability to represent the overall © 2023 "Copyright © 2023 for this paper by its authors. Use permitted under Creative Commons quality measure of a system based on four sub-attributes CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g LCicEenUseRAttWribuotironk4s.0hIontpernPatrioonacl e(CeCdBiYn4g.0)s." (CEUR-WS.org) including security, safety, reliability, and maintainability.
Accuracy Privacy Robustness Efficiency Effectiveness Transparency Reliability Safety Precision Security Sustainability
itself, but also from the process (how the product was
made), the tools and infrastructure (with what), the
people (by whom) as well as the governance (who decides).
Trustworthiness can also be considered from a quality
point of view or a risk point of view. In the former, the
chances to meet the stakeholder expectations are
maximized (by good practices and clear metrics). In the latter,
the chances not to meet the stakeholder expectations
are minimized (by identifying and mitigating potential
issues). Thus, all trustworthiness attributes can generally
be considered from a quality or risk point of view.
Quality is at the center of the SQuaRE (Systems and software
Quality Requirements and Evaluation) series of standards
ISO/IEC 25000:2014 and AI quality is more specifically
considered in ISO/IEC DIS 25059 (under development).
The quantification of AI-based system trustworthiness
has become a hot topic [12]. From a strict
metrological point of view, measurement is relative to a physical
property which can be compared to a reference quantity
of the same kind. Following this definition,
trustworthiness cannot be “measured”. However, each attribute
related to trustworthiness can be represented on a scale
(e.g., number scale, nominal scale, ordinals scale) [
Our approach is based on the following steps (see fig.2):
2. Step 2: Identification of numerical evaluations; 3. Step 3: Adapting attributes for commensurability; 4. Step 4: Definition of an aggregation methodology to capture operational trade-ofs and evaluate higher-level attributes.
Multi-Criteria Decision Aiding (MCDA) is a generic term for a collection of systematic approaches developed specifically to help one or several Decision Makers (DM) to assess or compare some alternatives on the basis of several criteria [13]. The dificulty is that the decision criteria are frequently numerous, dependent and sometime 2.1. Step 1: Semantic tree conflicting. For example, efectiveness may be conflicting with robustness, explainability, or afordability. The Based on diferent sources (norms, standards, scientific viewpoints are quantified through attributes (see §2.1). communications, industrial and institutional reports,
First, to assess AI trustworthiness, the choice of the rel- Confiance.ai reports, etc.), the characterization and evaluevant attributes is not easy, since the selection pertains to ation of trust attributes focus on defining and structuring the context of application, which is modeled according to the attributes that constitute trust in the context of AIseveral elements (Operational Design Domain, intended based safety critical systems [15] going beyond a risk domain of use, nature and roles of the stakeholders, etc.) analysis as proposed in [16, 17].
The attributes can be quantitative (typically numerical Our problem of assessing Trustworthiness is decomvalues either derived from a measure or providing a com- posed in several sub-problems by introducing a hierarprehensive and statistical overview of a phenomenon) or chy of an important number of specific criteria. This qualitative (based on the detailed analysis and interpreta- structuring phase aims to construct a tree representing a tion of a limited number of samples). Then once the list hierarchy of points of view in which the root represents of relevant attributes has been defined, the aggregation the overall evaluation, and the leaves are the elementary of several attributes remains complex due to commensu- attributes. In order to produce such a hierarchy, one shall rability issues: indeed, this is equivalent with combining succeed in grouping the criteria according to a classifica“oranges and apples”, none of the attributes having the tion that makes sense for the stakeholders. At the end of same unit. In addition, one aims at making trade-ofs this step, one shall obtain the relevant criteria together and arbitration between the attributes. This means that with their organization in a tree. This first step has been the value of each attribute should be transformed into a captured in the mind-map of Fig.3. scale common to all attributes and representing the pref- The attributes are currently grouped according to the erences of a stakeholder, and that the values of the scales capabilities they characterize: technology, ethics, interfor the diferent criteria should be aggregated. These ele- action and trust intermediaries (such as certification). ments constitute the main steps for solving the problem Technology is system-centric, it refers to the ability using an MCDA approach. to verify that the AI-based component has valid and ro
Aggregation functions are often used to compare al- bust intrinsic properties such as accuracy, robustness, ternatives evaluated on multiple conflicting criteria by safety and security. Thus, AI-based systems should gensynthesizing their performances into overall utility val- erate accurate output as consistent as possible with the ues [14]. Such functions must be suficiently expressive ground truth. Additionally, AI systems should be roto fit the DM’s preferences, allowing for instance the bust to changes, specifically in complex, dynamic and determination of the preferred alternative or to make uncertain real environments. Moreover, AI programs compromises among the criteria - improving a criterion or systems must not harm any human being under any implies that one shall deteriorate on another one. MCDA circumstances that prioritize user safety. In addition, provides a tool to specify the good compromises [13]. the autonomy of trustworthy AI should always be under user’s control. In other words, it has always been the human right to give the AI system decision-making representation and reasoning used in symbolic AI. authority or to revoke that authority at any time.
From interaction’s perspective, trustworthy AI 2.2. Step 2: Numerical evaluations should possess the properties of usability, explainability and interpretability. Specifically, AI-based systems All nodes return a numerical evaluation. Specific Key Pershould not cease operation at inappropriate times (e.g. at formance Indicators (KPI), metrics or evaluation methods times when the lack of output could lead to safety risks), are used to qualify the leaves of the tree according to the and these programs or systems should be easy to use for use cases. For example, data quality is a problem that people with diferent backgrounds. Trustworthy AI solu- has been studied for several decades now [19]. However, tions should allow explanation and analysis by humans primarily the focus has been on the data in operational to reduce potential risks and harms and empower human databases and data warehouses. Now, Data-driven AI users. In addition, trustworthy AI should be transparent is generating renewed interest in data quality, but there so people can better understand its mechanism. is yet no consensus on what comprises the data quality
Ethics, strongly linked in Europe to the notion of fun- characteristics. Thus, [20] were among the first argudamental rights, is notably put forward in the work of the ing that limiting quality to the level of accuracy is not AI HLEG (High-Level Expert Group on Artificial Intelli- enough, highlighting that the level of quality for given gence) of the European Commission [18]. A system must, data can depend on its purpose. Its principles require an for example, be law-abiding, fair, accountable, environ- assessment of the various quality attributes as presented mentally friendly and compliant with the user privacy. in §3.2, mainly in fig.6. Standards are currently being deSpecifically, AI systems should operate in accordance veloped to define data quality attributes for ML (Machine with all relevant laws and regulations, as well as with the Learning): ISO/IEC CD 5259-1 (terminology and princiethical principles of human society. ples) and ISO/IEC CD 5259-2 (data quality measures).
As some notions (e.g. explainability) concern several
dimensions (ethics vs. interaction), Confiance.ai pro- 2.3. Step 3: Commensurability
gram made an arbitrary choice to be consistent with the
methodology. Finally, the attributes for trusted ecosys- Aggregating diferent attributes for a global assessment
tem intermediaries focus on the relationships to third- requires that they are commensurate. This implies that
parties, in particular quality assurance, audit and cer- one shall be able to compare any numerical evaluation
tification activities. All these properties apply to the of an attribute with any numerical evaluation of any
AI-based component and the system that embeds AI, but other attribute. In order to make the assessment
"comthey also apply to the quality of the data used for train- parable," sound methods for normalization (to make the
ing connectionist AI and/or to the quality of knowledge comparison between variables comparable) have to be
applied to single variables in order to first make them gation function where (0, ..., 0) = 0, (1, ..., 1) = 1,
comparable, that is, transforming the various scales of and (1, ..., ) ≤ (1, ..., ) if ≤ , ∀.
Morevariables into one unique scale. The numerical evalua- over, the utility function normalizes the metrics and
protion of the attributes is thus encoded in the [
is only a small part of the overall system and significant additional engineering and system functionalities are required to ensure that the model can operate in a reliable, predictable and scalable way with proper engineering of data and model pipelines, monitoring and logging, etc.
While the necessity and usefulness of reasoning about trust assessment is obvious, obtaining trustworthiness scores remains a challenging task. To illustrate such issues, as stated previously, some aspects linked to trustworthiness are highly subjective or context dependent.
For example, the notion of “data quality” (resp. “robustness”, “explainability”) requires having a knowledge of Figure 4: Radar chart is useful for comparing two systems all induced attributes including those that are system which embed diferent AI approaches. dependent such as data availability, data portability, data precision, etc. (resp. adaptability, durability, resilience,
Radar chart is a visual method for comprehensive eval- etc.). The subjectivity or vagueness of the attribute defuation, particularly useful for holistic and overall assess- initions does not always represent a major hindrance ment through multivariate data. However, this represen- to use them in operational settings, because skills and tation does not allow understanding the interactions and knowledge of AI and safety engineers may be enough dependencies between attributes. to determine what may be appropriate thresholds and
The goal of MCDA is to define a numerical represen- scores.
tation of the preferences of the stakeholders, expressed
as a function : → [
The assessment of the data timeliness attribute [25] indicates whether the data was submitted in due time, respecting the data gathering deadline: _ = 1 ︂( − age of the data value)︂ shelf life with two dimensions showing actual values and predicted values. Each row of the confusion matrix represents the instances in a predicted class and each column represents the instances in an actual class. Each cell in the confusion matrix represents an evaluation factor. For example, for a binary classification of "positive" and "negative": samples your model predicted correctly. ∙ True Positive ( ) signifies how many positive class samples your model predicted correctly. ∙ True Negative ( ) signifies how many negative class samples your model predicted incorrectly. ∙ False Positive ( ) signifies how many positive class samples your model predicted incorrectly. ∙ False Negative ( ) signifies how many negative class A precision score (see fig. 5) close or equal to
1 will signify that your model did not miss any true positives, and is able to classify well between correct and incorrect labeling of observed data. Recall is the proportion of actual positives that the model has correctly identified as actual negatives that the model has correctly identified as such out of all negatives. A high F1 score symbolizes a high precision as well as high recall. It presents a good balance between precision and recall and gives good results on imbalanced classification problems. The
ROC Curve is a plot which shows the performance of a binary classifier as function of its cut-of threshold. It essentially shows the rate against the rate for various threshold values. Selecting the most suitable evaluation metric strongly depends on the way how the stakeholder defines the criticality of the application.
Data quality is at the center of the standard ISO/IEC 25012:2008. The standard distinguishes between “nherwhere “age of the data value” represents the time diference between the occurrence (i.e., when the data value was created) and the assessment of timeliness of the data value; “shelf life” is defined as the maximum length of time the values of the considered attribute remain up-todate, which can be determined through expert knowledge.
higher value of the metric for timeliness, and vice versa.
on expert estimations, influences the sensitivity of the metric to the ratio (age of the data value / shelf life).
on the Ge and Helfert’s ratio [26], and defined as: _ = ∑︀ 0 if is a missing data, and 1 otherwise.
=1 ()/ where () is Data correctness could be defined as: _ =
1/(1 + (, )) where is the data value to be assessed, s the corresponding real value and is a domain-specific distance measure
larger value of the distance function, which in turn leads to a larger denominator and thus a smaller metric value.
The IEEE glossary of software engineering [27] defines robustness as “The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions”. Related terms are thus error and fault tolerance, the first one regarding error in data inputs whereas the second at component level. In the present context, the robustness of an AI-enabled system essentially depends and is focused on the ML or DL (Deep Learning) components and the phase in the development cycle where the training model is designed and such out of all positives. Specificity
is the proportion of such as the Euclidean distance or the Hamming distance. ent data quality” and “system-dependent data quality”. tested (i.e. the phase involving ML/DL algorithms, trainThe former is intrinsic to the data and does not depend ing and testing data sets). As it has been highlighted in on the application (e.g. correctness). The later is appli- [28], ML/DL models exhibit counter-intuitive properties cation specific (e.g. accuracy). Data quality can also be considered for a complete data set (e.g. completeness) or for a unique value (e.g. currentness). For example, some data quality characteristics are described in fig. 6.
Traditionally, metrics for data accuracy are based on the rate of correct data items over an entire data set, like (1) the misclassification of (adversarial) perturbations that are statistically indistinguishable from the ones in the training data set (e.g. identically distributed noises) and (2) the misclassification of data subsets representing a semantic unit/object in the presence of minor perturbations that break data regularity whereas still being using a 1 for an accurate data item, and a 0 otherwise: readable by humans (e.g. small square in stop signal).
Characteristic
Accuracy Completeness Consistency Correctness
Currentness Representativeness
Definition The degree to which the data has attributes that correctly represent the true value of the intended attribute of a concept or event in a specific context of use.
The degree to which subject data associated with an entity has values for all expected attributes and related entity instances in a specific context of use.
The degree to which data has attributes that are free from contradiction and are coherent with other data in a specific context of use.
Degree to which the data is free from errors.
The degree to which data has attributes that are of the right age in a specific context of use.
Degree to which the data is representative of the statistical population. Overall, a definition allowing to test the robustness of a fication of the same points after adding new training data model is the measure of the impact of the minimum and updating model parameters, etc. Such apparently adversarial perturbation across many samples . In ad- inconsistent outcomes make designers raise questions dition, robustness can (should) be tested at two levels of about ML/DL models’ foundations and call for methods to possible perturbations as follows [29]: characterize them. Thus, explainability is a term used to Local robustness is satisfied by a single data input ∈ encapsulate and refer to all previous needs and aims and of a model and a given perturbation ′ within a is therefore a qualitative attribute of AI-based systems. neighborhood if () is identical to (′), in other That said, there are no unified methods or scales to evalwords: ∀′, (, ′) ≤ ⇒ () = (′) uate explainability. Recent surveys, as the one ofered by Global robustness is satisfied by the set of data of [30], suggest that explainability can be decomposed by a model , considering possible perturbations ′ for the methods used to evaluate it. A brief description of all inputs ∈ , and exhibiting smooth convergence the main families found in literature is provided below. of (′) towards () during classification, in other Visualization methods pursue the characterization words: ∀, ′ ∈ , (, ′) ≤ ⇒ () → (′). of a ML/DL network by visual observation of the levels of If the model outputs () conform a dense set allowing activation/deactivation according to the input data and a distance metrics (.), the convergence can be validated their influence in the classification performance, sensifor a given > 0 satisfying ( (), (′)) < . In tivity, and other functional/structural properties. Reprepractice, such post-condition could be dificult or un- sentative instances in this family are: feasible to verify depending upon the nature of (). Back-propagation helps to observe relevance of data in Further means are thus needed to understand how per- terms of the activation/deactivation gradients observed turbations impact misclassification. at diferent layers in the network during training, e.g. Activation Maximization [31], Deconvolution [32], Layer3.4. Explainability wise Relevance Propagation [33].
Perturbation-based methods provide means to observe Explainability is by far one of the most rich and complex and compare its impact in the network w.r.t. nonto assess feature in recent research concerning ML/DL perturbed input, e.g. Occlusion Sensitivity [32], Repretopics. Several reasons justify such fact, in particular sentation Erasure [34], Meaningful Perturbation [35]. because explainability aims to provide answer as to why Distillation methods aim to represent (distill) the ML/DL algorithms succeed or fail, which is rather a chal- knowledge encoded in the ML/DL network after trainlenge given their heuristic nature and intricacy. Explain- ing via a more human-readable format suitable for both ability is also a high-level demand in several domains user interpretation and logic/machine reasoning. Some aiming to transfer safety-critical human-based tasks to representative instances in this family are: autonomous systems, e.g. for accountability purposes, Local Approximation methods mimic the input/output a basic explainability requirement for a self-driving ve- behavior of the target ML/DL model on smaller data sets, hicle being to suficiently characterize the contribution and using approximation functions, e.g. linear functions. of the ML/DL network branches during pedestrian/ob- Local Approximation methods are based upon the hystacle detection. Last, yet not the least, explaining the pothesis that the ML/DL behavior can be better and more behavior of ML/DL models can be tihgtly coupled to easily characterized on local areas rather than over the solve apparent conflicts/inconsistencies between certain entire data set, e.g. LIME [36], Anchors [37]. attributes/features. To illustrate this point, we recall the Model Translation methods aim to mimic input/output counter-intuitive properties of a ML/DL model referred in behavior of the target ML/DL model however considering [28] which are rather related to robustness, e.g. misclassi- the whole data set over a symbolic model, e.g. Graphifcation of statistically indistinguishable points, misclassi- based [38], Rule-based [39].
explainability as part of the design of the ML/DL model.
The explainability of ML/DL networks should be intrinsic and thus input/output behavior should be explicitly justiifed by the ML/DL model itself. Representative instances in this family are: Attention Mechanisms rely upon contextual vector and attention mechanisms used to learn a conditional distribution over data inputs which provide an interpretation on the behavior of the weights of the operations of activation and deactivation, e.g. Single Modal Weighting [40], Multimodal Interaction [41].
Joint Training consists in introducing an additional task in the ML/DL model, asides the original one, in charge of providing direct or indirect explanations for the main task behavior, e.g. Text Explanation [42], Explanation Association [43].
Being a qualitative feature, explainability in turn requires criteria to evaluate the quality of explanations.
This presupposes a non-negligible intervention of humans in the assessment process. Some methods proposed to evaluate explanations can be found in [30]. criteria decision aid, Annals of Operations Research Technical Report, Départment d’Informatique et 175 (2010) 247–286. Recherche Opérationnelle, Univ. of Montreal, 2009. [15] L. Pons, I. Ozkaya, Priority quality attributes for [32] M. D. Zeiler, R. Fergus, "visualizing and understandengineering AI-enabled systems, arXiv:1911.02912 ing convolutional networks", in: D. Fleet, T. Pajdla, (2019). B. Schiele, T. Tuytelaars (Eds.), "Computer Vision [16] High-Level Expert Group on Artificial Intelligence, – ECCV 2014", Springer International Publishing, Assessment List for Trustworthy Artificial Intelli- 2014, pp. 818–833. gence (ALTAI), Technical Report, European Com- [33] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, mission, 2019. K.-R. Müller, Explaining nonlinear classification [17] D. Piorkowski, M. Hind, J. Richards, Quantitative decisions with deep taylor decomposition, Pattern ai risk assessments: Opportunities and challenges, Recognition 65 (2017) 211–222.
arXiv preprint arXiv:2209.06317 (2022). [34] J. Li, W. Monroe, D. Jurafsky, Understanding neural [18] A. HLEG, Assessment list for trustworthy artificial networks through representation erasure, CoRR intelligence (altai) for self-assessment, High Level abs/1612.08220 (2016).
Expert Group on Artificial Intelligence. B-1049 Brus- [35] R. C. Fong, A. Vedaldi, Interpretable explanations sels (2020). of black boxes by meaningful perturbation, in: 2017 [19] J. Mattioli, P.-O. Robic, E. Jesson, Information qual- IEEE International Conference on Computer Vision ity: the cornerstone for AI-based industry 4.0, Pro- (ICCV), 2017, pp. 3449–3457.
cedia Computer Science 201 (2022) 453–460. [36] M. T. Ribeiro, S. Singh, C. Guestrin, "why should i [20] R. Y. Wang, D. M. Strong, Beyond accuracy: What trust you?": Explaining the predictions of any clasdata quality means to data consumers, Journal of sifier, in: Proceedings of the 22nd ACM SIGKDD management information systems 12 (1996) 5–33. International Conference on Knowledge Discovery [21] C. A. Bana e Costa, J.-C. Vansnick, A theoretical and Data Mining, Association for Computing Maframework for Measuring Attractiveness by a Cate- chinery, New York, NY, USA, 2016, p. 1135–1144. gorical Based Evaluation TecHnique (MACBETH), [37] M. T. Ribeiro, S. Singh, C. Guestrin, Anchors: Highin: Proc. XIth Int. Conf. on MultiCriteria Decision precision model-agnostic explanations, in: ProMaking, 1994, pp. 15–24. ceedings of the AAAI conference on artificial intel[22] C. A. Bana e Costa, M. Oliveira, A multicriteria deci- ligence, volume 32, 2018, p. 1527–1535. sion analysis model for faculty evaluation, Omega [38] Q. Zhang, R. Cao, F. Shi, Y. N. Wu, S.-C. Zhu, Inter40 (2012) 424–436. preting cnn knowledge via an explanatory graph, [23] G. Choquet, Theory of capacities, in: Annales de in: Proceedings of the AAAI Conference on Artifil’institut Fourier, volume 5, 1954, pp. 131–295. cial Intelligence, volume 32, 2018, pp. 4454–4463. [24] L. Sun, H. Dong, A. X. Liu, Aggregation func- [39] M. Harradon, J. Druce, B. E. Ruttenberg, Causal tions considering criteria interrelationships in fuzzy learning and explanation of deep neural networks multi-criteria decision making: state-of-the-art, via autoencoded activations, CoRR abs/1802.00541 IEEE Access 6 (2018) 68104–68136. (2018). [25] C. Batini, M. Scannapieco, Data and Information [40] L. A. Hendricks, Z. Akata, M. Rohrbach, et al., Gen
Quality, Springer International Publishing, 2016. erating visual explanations, in: B. Leibe, J. Matas, [26] M. Ge, M. Helfert, A framework to assess decision N. Sebe, M. Welling (Eds.), Computer Vision – quality using information quality dimensions., in: ECCV 2016, Springer International Publishing, 2016, ICIQ, 2006, pp. 455–466. pp. 3–19. [27] ANSI/ IEEE Std 729-1983, Ieee standard glossary of [41] P. Anderson, X. He, C. Buehler, et al., Bottom-up software engineering terminology, 1983. and top-down attention for image captioning and [28] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Er- visual question answering, in: 2018 IEEE/CVF Conhan, I. Goodfellow, R. Fergus, Intriguing properties ference on Computer Vision and Pattern Recogniof neural networks, arXiv:1312.6199 (2013). tion, 2018, pp. 6077–6086. [29] J. M. Zhang, M. Harman, L. Ma, Y. Liu, Machine [42] H. Liu, Q. Yin, W. Y. Wang, Towards explainable learning testing: Survey, landscapes and horizons, NLP: A generative explanation framework for text IEEE Transactions on Software Engineering 48 classification, CoRR abs/1811.00196 (2018). (2022) 1–36. [43] R. Iyer, Y. Li, H. Li, M. Lewis, R. Sundar, K. Sycara, [30] N. Xie, G. Ras, M. van Gerven, D. Doran, Explain- Transparency and explanation in deep reinforceable deep learning: A field guide for the uninitiated, ment learning neural networks, in: Proceedings CoRR abs/2004.14545 (2020). of the 2018 AAAI/ACM Conference on AI, Ethics, [31] D. Erhan, Y. Bengio, A. C. Courville, P. Vincent, Vi- and Society, Association for Computing Machinery, sualizing Higher-Layer Features of a Deep Network, 2018, p. 144–150.