Methods from explainable machine learning are increasingly applied. However, evaluation of these methods is often anecdotal and not systematic. Prior work has identified properties of explanation quality and we argue that evaluation should be based on them. In this position paper, we provide an evaluation process that follows the idea of property testing. The process acknowledges the central role of the human, yet argues for a quantitative approach for the evaluation. We find that properties can be divided into two groups, one to ensure trustworthiness, the other to assess comprehensibility. Options for quantitative property tests are discussed. Future research should focus on the standardization of testing procedures.
The development and adoption of complex machine learning (ML) models has given rise to
research that seeks to make these models explainable [
Evaluation of explainable systems is typically categorized into three types: (1)
applicationgrounded evaluation with humans and applied tasks (2) human-grounded evaluation with
humans but proxy tasks and (3) functionally-grounded evaluation without humans and proxy
tasks instead [
In a recent review of more than 300 papers from prominent ML conferences, it was found that
evaluation is roughly 30% anecdotal evidence and roughly 20% with users [
None of the goals of explainable ML, be it generating scientific insight or enabling model
verification, can be achieved on the basis of faulty explanations. Hence, it is crucial to ensure
that explainability systems deliver a high quality output that represents the model behavior
truthfully [
Not all properties are applicable to all explanation methods. For example, explanation methods
that involve random sampling are naturally less consistent than deterministic methods. One
idea is to use fact sheets to track applicable properties of an explanation method [
We have the impression that existing evaluation measures for UIPs are not widely used yet, but hope that the integration of measures into emerging coding-libraries1 has an efect on practice. 3.2. Ensuring Comprehensibility: User Dependent Properties (UDPs) Carrying out user studies is a costly and time-consuming process. We argue that generating reports on UDPs should be made as easy as possible for practitioners. We thus want to discuss possibilities to facilitate quantitative measurements of UDPs.
Compactness is a property that is closely tied to managing users’ Cognitive Load [17]. For this, we can turn to the concept of Working Memory [18] that describes the limit of how many chunks (meaningful items/ thoughts) a human can actively entertain at the same time. What 1e.g. Captum and Quantus constitutes a chunk in the respective application needs context-level definition. For text data it might be a single word, for image data it might be all pixels belonging to the same object. What constitutes a chunk also depends on user experience in the respective domain, e.g. a more experienced user having developed a more complex mental model to associate information with can process larger amounts of information than inexperienced users.
Plausibility measures to what extent the explanation corresponds to user expectations. Providing practitioners with a ground truth dataset containing human verified explanations would enable them to report a degree of explanatory overlap as explored by [16]. Remarkably, the study also found user biases in the subjective rating of explanations. This indicates that purely qualitative evaluation is not suficient providing another reason to develop quantitative measures. A possible disadvantage of ground truth datasets is that they are task specific, hence, transferability might be an issue. Another direction for plausibility checks are test scenarios, inspired by software engineering practices [ 19]. Test scenarios can be generated according to user expectations. In a scenario with text data and feature highlighting, possible checks could be the following: ”words with low information value, e.g. the, should not be part of an explanation” or ”replacing adjectives with synonyms should yield the same attribution”. However, usefulness of ground truth datasets or other test scenarios have their limits. If an explanation does not comply with user expectations that does not mean that it is not a faithful description of model behavior. Discrepancies between expected explanation and observed explanations have to be investigated carefully as to not blame the explainability method when it is really the model that behaves unexpectedly. If the explanation is trustworthy, a plausibility check does not reveal anything about the explanation but only something about the model.
Usability is the most elusive property of the three because it not only depends on the explanation methods used but also on presentation, UI, tech-afinity of the user and the specific questions the user wants to be answered. Options to quantitatively measure usability are timeon-task or task-success with an explanation condition (with or without explanation) [20, 21, 22]. The challenge remains that user studies, whether they are qualitative or quantitative, require time and resources.
This list of properties is non-exhaustive and relevance of individual metrics may vary depending on the application setting or the diversity of the targeted user group.
To establish good evaluation practices we propose the evaluation process shown in Figure 1. The key idea of this process is to enforce UIP evaluation both before evaluating UDPs and also after any explanation changing alterations to the system.
At first the Context needs to be captured [23]. User Interviews provide an investigation into what goals the users want to accomplish and what questions need to be answered, e.g. [24]. Beyond immediate user intent, engineers may rely on established Design Guidelines for comparable scenarios to inform initial design decisions, e.g. regarding compositionality.
This informs the Requirement Catalogue used to determine what Explainability Methods should be selected to fulfill identified user needs. Upon implementation, the system is evaluated with regards to applicable UIPs. If UIPs indicate an improper representation of model behavior
Design Guidelines
User Interviews
Requirement Catalogue Context
Explainability
Methods
User Independent
Property Checks
User Dependent Property
Checks Trustworthy
Comprehensible
Explanation
System by the explanations, the engineer returns to choosing diferent Explainability Methods and UIP checks are repeated. Once UIP checks are passed, the system is assessed with regards to the UDPs. This can involve a real user study or a quantitative evaluation. If the explanations are deemed incomprehensible, this requires an update of the Requirement Catalogue. Thus, UDPs can impact UIPs: For example, if tests failed for Compactness and the user cannot cope with the amount of information presented, displaying less information has an impact on Completeness of the displayed explanation. If the system is adapted, a re-evaluation of the UIPs is necessary to ensure trustworthiness is maintained.
Most likely, UIP-UDP-trade-ofs will be unavoidable in practice, e.g. due to specific user needs or a lack of choice of better explanation methods. Finding a balance will be dificult. Returning to our previous example: A UDP check indicates that the user prefers a more compact explanation. The engineer implements a slider to adjust the amount of chunks to be displayed at the same time. Seemingly, all information is still available to the user, yet the user can now pick at what level of compactness the explanation appears the most plausible. This rather enables them to confirm their own bias of the model instead of investigating the true model behavior. Providing the user with information about how their actions afect trustworthiness might be a possible remedy. However, in some scenarios, e.g. where an explanation system is used to investigate a model for certification or in a legal dispute, trustworthiness demands may be nonnegotiable.
In general, we see the risk that compromising UIPs, such as faithfulness, opens the way for
manipulation in adversarial settings [25]. We expect that defining the boundaries of
permissible UIP-UDP-trade-ofs will be challenging as indicated by literature discussing
faithfulnessplausibility trade-ofs [
For works that propose a novel explanation method without a direct use case, it would be helpful to provide information on applicable UIPs and a discussion of how UDPs might impact the performance under which circumstances.
”The food was delicious and the price somewhat fair .” If a user argues for more compactness the explanation could look like the following sentence with ”fair” not being highlighted. ”The food was delicious and the price somewhat fair.” By removing the term ”fair” from the feature highlighting the explanation is more compact but less complete.
The standardization of UIPs and UDPs would enable the development of software-packages that automatically test for applicable properties given an explanation system.
The number of quantitative tests available for UDPs is still low which we attribute to the subjective nature of UDPs. Nevertheless, quantitative tests for UDPs will be helpful and reliable means to better compare methods.
Furthermore, we recommend that newly proposed explanation methods perform ablation studies to simulate possible application scenarios, or at least include discussions to analyze trustworthiness-comprehensibility trade-ofs. Likewise, indicating applicable UIPs and reporting on respective statistics should become the standard, similar to the reporting that is customary for predictive performance.
Trust is often stated as goal of explainable ML. Naturally, it would be interesting to investigate how explanation systems developed with the proposed process score with regards to (in-) appropriate trust.
Evaluation of explainable ML is essential to demonstrate trustworthiness and ensure comprehensibility of an explanation system. However, there are no guidelines or standards for evaluation yet. In this work, we argued for quantitative human-grounded evaluation of explainable ML. We presented an evaluation process that includes the user and places importance on quantitative tests. Explanation quality is considered as a construct consisting of several properties. We split the properties into two sets, one to ensure trustworthiness, the other to assess comprehensibility. We highlight that adaptations of the explanation system based on comprehensibility checks can impact properties that quantify trustworthiness. We raise the question whether this trade-of is justifiable under any circumstances and call for a discussion within the community to further develop a position.
We thank the ML2R Trustworthy ML group and the MLAI Lab for the helpful discussions. This research has been partly funded by the Federal Ministry of Education and Research of Germany as part of the competence center for machine learning ML2R (01|S18038BC). K. Beckh has contributed as part of the SmartHospital.NRW project which is funded by the Ministry for Economic Afairs, Innovation, Digitalization and Energy of the State of North Rhine-Westphalia. [16] S. Mohseni, J. E. Block, E. Ragan, Quantitative evaluation of machine learning explanations: A human-grounded benchmark, in: 26th International Conference on Intelligent User Interfaces, 2021, pp. 22–31. [17] J. L. Plass, R. Moreno, R. Brünken, Cognitive load theory, Cambridge University Press (2010). [18] N. Cowan, The magical mystery four: How is working memory capacity limited, and why?, Current Directions in Psychological Science 19 (2010) 51–57. [19] M. T. Ribeiro, T. Wu, C. Guestrin, S. Singh, Beyond accuracy: Behavioral testing of NLP models with CheckList, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, Online, 2020, pp. 4902–4912. [20] I. Lage, E. Chen, J. He, M. Narayanan, B. Kim, S. J. Gershman, F. Doshi-Velez, Human evaluation of models built for interpretability, in: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, 2019, pp. 59–67. [21] J. P. Dietvorst, Berkeley J .and Simmons, C. Massey, Algorithm aversion: People erroneously avoid algorithms after seeing them err, Journal of Experimental Psychology: General 144 (2015) 114. [22] F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hofman, J. W. Wortman Vaughan, H. Wallach, Manipulating and measuring model interpretability, in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, ACM, 2021. [23] U. Ehsan, Q. V. Liao, M. Muller, M. O. Riedl, J. D. Weisz, Expanding explainability: Towards social transparency in ai systems, in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–19. [24] Q. V. Liao, D. Gruen, S. Miller, Questioning the AI: Informing Design Practices for Explainable AI User Experiences, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020) 1–15. URL: http://arxiv.org/abs/2001.02478. arXiv:2001.02478. [25] S. Bordt, M. Finck, E. Raidl, U. von Luxburg, Post-hoc explanations fail to achieve their purpose in adversarial contexts, arXiv preprint arXiv:2201.10295 (2022). [26] Z. C. Lipton, The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery., Queue 16 (2018) 31–57.