The importance of explanations in decision-making, particularly in the medical domain, has been widely recognized. However, the evaluation of the quality of these explanations remains a challenging task. In this work, we propose a novel approach for assessing and evaluating the reasons provided in explanations about clinical cases. Our approach leverages an external knowledge base and a defined prevalence function to score each reason based on its pertinence in the domain. By applying a deterministic prevalence function, we ensure total transparency of the reasons' assessment, facilitating a precise explanation of the rationale behind the scoring hierarchy of each reason. We demonstrate the efectiveness of our approach in clinical cases, where medical experts explain the rationale behind a specific diagnosis and why other potential diagnoses are dismissed. Our methodology provides a nuanced and detailed evaluation of the explanation, contributing to a more comprehensive understanding of the decision-making process.
Explainable Artificial Intelligence (XAI) [
To address this challenging open issue, we propose a novel approach for an assessment and evaluation of the reasons employed in explanations, which satisfies transparency as well. More precisely, our goal is to automatically evaluate the relevance of all conceivable reasons that could explain a particular event, and subsequently compare them with the reasons invoked by the explainer.
When applied to the medical field, our approach scrutinizes explanations provided in medical
examinations, wherein medical residents elucidate a specific diagnosis of a patient, given
the context (i.e., a clinical case detailing the patient’s condition) and their medical expertise.
Consequently, we generate an assessment that identifies the reasons employed in the explanation
and evaluates them against the relevance scoring produced by our approach. Our approach
leverages an external knowledge base, the Human Phenotype Ontology (HPO) [
While our methodology is assessed on a use case from the medical domain, it is abstract enough to be applied to any domain. We envision two potential scenarios where our methodology could be particularly beneficial: AI for education and online medical fora.
In the context of AI for education, our approach can assist medical resident students in learning how to solve medical cases and develop a logical and explainable reasoning process in order to explain a diagnosis. By providing a systematic and transparent way of evaluating the reasons given in explanations, we can help students understand the rationale behind a specific diagnosis and why other potential diagnoses are dismissed.
In online medical fora, our approach can help online users to distinguish good explanations from bad explanations present there. Users often discuss diagnoses and share their experiences, but the quality of these discussions can vary widely. With our approach, we can provide a systematic and transparent way of evaluating the reasons given in these discussions, helping users and moderators identify high-quality explanations and promote more informed discussions.
The research presented in this paper is driven by the necessity for a systematic and transparent methodology to assess the pertinence of reasons used in medical explanations. To the best of our knowledge, this is the first approach that leverages an external knowledge base, the Human Phenotype Ontology (HPO), and a deterministic prevalence function to evaluate the reasons for the potential diagnoses based on their relevance in the context of a specific clinical case and grounding on the HPO knowledge base.
The paper is organized as follows: after a comparison with the related work, we first describe our data preprocessing heuristics (Section 3.1), followed by the extraction and encoding of reasons from the clinical case into HPO terms (Section 3.2). Next, we describe the computation of the pertinence score for each reason using the prevalence function (Section 3.3) and discuss its deterministic nature. Then, we illustrate the sentence-matching approach employed to align the extracted reasons with those found in the explanation (Section 3.5). Finally, we demonstrate the generation of a pertinence assessment of the reasons following a template-based generation technique (Section 3.6).
In this section, we discuss the related work on explanation selection and then we focus on the medical domain application scenario.
Explanation Selection. In the process of explanation selection, individuals choose what they
perceive to be the most relevant causes from a larger set of causes for a particular event. This
selection is not arbitrary and is guided by criteria such as temporality, abnormality, intention,
and the diferences between a fact and a foil [
XAI for the medical domain. The importance of explanations in AI systems, particularly
in the medical domain, has been extensively studied [
Our approach to assessing the reasons used in explanations is visualized in Figure 1. We start with a clinical case of a patient, supplemented by an explanation provided by a medical expert, which elaborates on the specific diagnosis attributed to the patient. The objective is to evaluate the identified reasons in the clinical case and the external knowledge, and compare them with the reasons invoked by the expert to justify the medical diagnosis. To achieve this, we compute a pertinence score using a deterministic prevalence function (see Section 3.3), which ensures complete transparency, allowing us to explain why each reason is more or less pertinent than the others with respect to the given case.
The HPO Ontology [
Our approach consists of two main steps: (i) the reasons given in the explanation are extracted
from the clinical case and encoded into HPO terms (following the approach of Marro et al. [
The foundation for our work is the Antidote Casimedicos dataset [
This dataset is unique in the medical domain as it consists of 621 theoretical, fictional clinical cases that provide real expert-crafted explanations, making it a valuable resource for students. Each clinical case is paired with a set of potential diagnoses, an indicator of the correct answer, and a detailed explanation of the decision-making process provided by medical professionals2. What sets this dataset apart from other Question Answering (QA) datasets in the medical domain is its inclusion of explanatory arguments for both the correct diagnosis or treatment and the reasons why other options are incorrect. These explanations, written by medical doctors, ofer a rich source of information for research in XAI.
To prepare the data, we first enhance contextual information in the explanations by expanding
abbreviated diagnosis references. It is common for the explainer to refer to diagnoses as “Answer
1”, thus we implement a string replacement with the corresponding answer. Subsequently,
as delineated in [
The initial phase of our approach (Figure 1) consists in identifying all potential causes within
the given context. In our medical scenario, the context is represented by clinical cases, and we
regard all symptoms as potential causes that could explain the patient’s diagnosis. To address
this, we utilize the clinical information extraction pipeline proposed by Marro et al. [
(i) Medical named entity recognition: Marro et al. first identify medical concepts and abnormalities described in the clinical case text using a named entity recognition (NER) system. Their NER module detects mentions of symptoms, findings, and other phenotypic concepts, labelling 2https://github.com/ixa-ehu/antidote-casimedicos them with semantic tags like "Sign or Symptom" or "Finding." Marro et al. trained this module on a dataset of 314 annotated clinical cases, achieving promising results for symptom detection with an F1-Score of 0.86 using a transformer-based model. The NER model can accurately label symptoms described in layperson terminology, an important capability since clinical cases are often written for student exams, where terms may not appear standardised.
(ii) Alignment to ontology terms: Next, Marro et al. map the detected semantic concepts to
the standardized terminology of the Human Phenotype Ontology (HPO) [
The output we obtain from their pipeline is a list of symptoms extracted from the case text and aligned to ontology concepts. We select the Sign or Symptom and No Sign or Symptom entities, as these denote the presence or absence of abnormalities relevant for diagnosis. This aligned set of reasons extracted from the specific clinical case serves as our starting point. We later expand this list by retrieving additional related symptoms from the HPO knowledge base.
Ultimately, Marro et al.’s techniques provide us with an initial set of ontology-grounded reasons consisting of the aligned symptoms and no symptoms identified within the context of the particular clinical case description. We subsequently feed this encoded evidence as input to the prevalence function, which will evaluate and score the relevance of these reasons from the case alongside other reasons derived from the external HPO knowledge.
The Prevalence Function3 is a central component of our approach, designed to systematically assess the pertinence of each possible reason that could explain a given event. This function is inspired by cognitive processes involved in explanation selection, aiming to replicate these processes in a transparent and replicable manner. The function takes into account the possible reasons found in the clinical case (see Section 3.2) and a KB that serves to find other possible reasons outside of the context but still relevant to the case. It then evaluates all possible reasons based on a set of predefined conditions, each of which contributes to the nfial prevalence score of the key reason. These conditions include whether the key reason is linked to the correct or incorrect diagnosis, its occurrence rate, and whether it is unique to the correct diagnosis, or shared among all possible diagnoses. In line with the abnormal conditions model, our function assigns a higher score to key reasons that are unique to the correct diagnosis and have a low occurrence rate. This reflects the idea that abnormal conditions, i.e., conditions that do not usually occur, are more likely to be the cause of an event. Moreover, our approach integrates the concept of contrastive explanation. For instance, if a symptom associated with an incorrect diagnosis has a high occurrence rate and does not appear in the clinical case, it can be invoked to discard the incorrect diagnosis. This aligns with the idea that the diferences between two events form the basis for the explanation.
The computation of the prevalence function starts with the acquisition of the set of potential reasons to be evaluated. In the context of medical diagnosis, these reasons correspond to 3The full system implementation will be available upon acceptance symptoms, which can be identified either within the clinical case or within the Human Phenotype Ontology (HPO) as symptoms associated with each potential diagnosis. The ontology provides not only a list of symptoms for each diagnosis but also pertinent information about each symptom, such as its occurrence rate, definition, and synonyms. This information facilitates the definition of three distinct sets of reasons, which serve as the basis for the computation of the Prevalence function: • : symptoms that belong to the correct disease; • : symptoms that belong to all the incorrect diseases; • : symptoms found in the case description.
The Prevalence Function is then used in conjunction with the additional disease information and symptom sets obtained from the HPO to produce a list of key reasons and their calculated prevalence scores. This allows us to provide a robust and transparent framework for assessing the quality of the reasons on which the explanations are grounded.
The Prevalence Score Function, as outlined in Algorithm 0, is designed to assess the relevance or pertinence of a given key reason in the context of a clinical case (CC) and a knowledge base (KB). The function operates by assigning a score to the key reason based on its presence in the correct and incorrect diagnoses, its occurrence rate, and its presence in the clinical case.
The function begins by initializing the score to zero and setting several boolean variables to false (lines 2-6). It then retrieves the symptoms associated with the correct and incorrect diagnoses from the knowledge base (lines 7-8) and identifies the symptoms present in the clinical case using Named Entity Recognition (NER) (line 9).
Then, it checks if the key reason is present in the symptoms of the correct diagnosis (lines 11-15). If it is, the function increments the score and sets the variable linkedToCorrectDiagnosis to true. If not, linkedToCorrectDiagnosis is set to false.
Next, the function checks if the key reason is present in the symptoms of the incorrect diagnoses (lines 17-20). If it is, the variable linkedToIncorrectDiagnosis is set to true. If not, it is set to false.
The function then checks the occurrence rate of the key reason (lines 22-29). If the key reason has a high occurrence rate (more than 70%), the variable hasHighOccurrenceRate is set to true, and if it is linked to the correct diagnosis, the score is incremented. If the key reason has a low occurrence rate (less than 30%), the variable hasLowOccurrenceRate is set to true.
It then checks if the key reason is unique to the correct diagnosis or shared with other diagnoses (lines 31-40). If the key reason is unique to the correct diagnosis and has a low occurrence rate, the score is incremented twice. If the key reason is shared with other diagnoses, the score is decremented.
Finally, the function checks if the key reason is present in the symptoms of the incorrect diagnoses but not in the present symptoms (lines 42-48). If the key reason has a high occurrence rate, the score is incremented. Otherwise, the score is decremented.
The final score represents the prevalence of the key reason in the context of the specific clinical case, providing a measure of its relevance or pertinence.
Algorithm 1 Prevalence Function 1: procedure PrevalenceFunction( , , , ) = 0 = ℎ ℎ = ℎ = ℎ ℎ = = ( ) = ( ) = ( ) if is in then = score = score + 1 , else
= end if if is in then
= else
= end if if has a high occurrence rate (more than 70%) then ℎ ℎ = if == then
score = score + 1 end if else if has a low occurrence rate (less than 30%) then
ℎ = end if if is in then if is not in then = = + 1 if ℎ == then
= + 1 end if else if is in then ℎ ℎ = = − 1 end if end if if is in then if is not in then if has a high occurrence rate then = + 1
1
A crucial step in our approach is the alignment of potential causes (i.e., reasons) identified in the clinical case with those actually invoked in the expert’s explanation. This alignment (visualized as the “Reasons alignment module” in Figure 1) is achieved through a sentence matching technique. The objective of this step is to discern which of the potential reasons identified were actually utilized by the experts in their explanation, thereby enabling subsequent suggestions of modifications to enhance the explanation’s pertinence.
Our approach to sentence matching is inspired by the work of Lu et al. [
Subsequently, we compute sentence embeddings using only the identified named entities.
This computation leverages the Sentence Transformers method [
In the final step of our pipeline, we employ a template-based generation approach to articulate the pertinence of each reason. This approach allows us to generate natural language explanations that are understandable by human users. Each template is designed to address a specific combination of features associated with a reason, and the appropriate template is selected based on the values of these features for each reason. The features considered in our approach are: • uniqueToCorrectDiagnosis indicates whether the reason is unique to the correct diagnosis. • sharedToOtherDiagnosis indicates whether the reason is shared with other diagnoses. • hasLowOccurrenceRate indicates whether the reason has a low occurrence rate. • hasHighOccurrenceRate indicates whether the reason has a high occurrence rate. • linkedToCorrectDiagnosis indicates whether the reason is directly linked to the correct diagnosis. • linkedToIncorrectDiagnosis indicates whether the reason is linked to an incorrect diagnosis. • presentInClinicalCase indicates whether the reason is present in the clinical case.
Based on the values of these features, a template, like the following, is selected to generate the explanation: • Template 1: "You should consider invoking the reason [reason] since it is unique to the correct diagnosis, has a high occurrence rate, and is present in the clinical case." • Template 2: "You should also consider invoking the reason [reason] since it is a symptom with a high occurrence rate for the incorrect disease [disease], and it does not appear in the clinical case, supporting discard [disease] as the correct diagnosis." • Template 3: "You should consider removing the reason [reason] since it is a common symptom alongside all possible diagnoses."
This template-based generation approach allows us to generate explanations that are informative and specific to the context of each reason, thereby ensuring the interpretability of the proposed approach.
In this section, we first present the experimental setting we propose to assess our approach, and then we discuss the obtained results. Finally, we apply our approach to a clinical case to discuss the final outcome of the pipeline.
The main experimental component of our task is the named entity-based sentence matching.
This task can be decomposed into two subtasks: first, the generation of tuples of similar
sentences, and second, the fine-tuning of Language Models (LMs) using the generated dataset.
For tuple generation, we implemented two approaches:
Baseline string distance method : As a baseline, we employed a simpler method based on
string distance between full sentences, similar to the work of Lu et al. [
Named entity-based method : Subsequently, we generated tuples using the medical-named
entities identified by Marro et al.’s pipeline [
For both cases, a dataset composed of
(_, _, _ , { | })
tuples is generated. For the fine-tuning of the LMs, we employ the PyTorch implementation
provided by Hugging Face [
In this section, we present the obtained results on the Casimedicos dataset. We evaluate the quality of explanations written by experts, highlighting both successful and unsuccessful examples. The performance of the sentence matching task is quantified in terms of macroaverage precision, recall, and F1 score, as shown in Table 1.
We adopt the all-mpnet-base-v2 model as our baseline, being it the state-of-the-art for
sentence embedding computation across various domains. However, our results indicate that
domain-specific models outperform this baseline across all metrics. In particular, the model
based on PubMedBert [
To illustrate the outcome of our approach, we present a full clinical case, the expert’s explanation, and the assessment of reasons from the CasiMedicos dataset. We consider a clinical case where the correct diagnosis is Porphyria cutanea tarda. The other potential diagnoses considered are Epidermolysis bullosa acquisita, Acute intermittent porphyria, and Ulerythema ophryogenesis. Clinical Case: “A 62-year-old man with a history of significant alcohol abuse, carrier of hepatitis C virus, treated with Ibuprofen for tendinitis of the right shoulder, goes to his dermatologist because after spending two weeks on vacation at the beach he notices the appearance of tense blisters on the dorsum of his hands. On examination, in addition to localization and slight malar hypertrichosis.” Expert’s Explanation: “Porphyria Cutanea Tarda: 60% of patients with PCT are male, many of them drink alcohol in excess, women who develop it are usually treated with drugs containing estrogens. Most are males with signs of iron overload, this overload reduces the activity of the enzyme uroporphyrinogen decarboxylase, which leads to the elevation of uroporphyrins. HCV and HIV infections have been implicated in the precipitation of acquired PCT. There is a hereditary form with AD pattern. Patients with PCT present with blistering of photoexposed skin, most frequently on the dorsum of the hands and scalp. In addition to fragility, they may develop hypertrichosis, hyperpigmentation, cicatricial alopecia, and sclerodermal induration.” Assessment of Reasons: The generated explanations for the top and bottom scoring reasons are as follows: • You should consider invoking the reason Elevated urinary delta-aminolevulinic acid since it is a symptom with a high occurrence rate for the incorrect disease Acute intermittent porphyria, and it does not appear in the clinical case, supporting discard Acute intermittent porphyria as the correct diagnosis. • You should also consider invoking the reason Abnormal hair morphology since it is a symptom with a high occurrence rate for the incorrect disease Epidermolysis bullosa acquisita, and it does not appear in the clinical case, supporting discard Epidermolysis bullosa acquisita as the correct diagnosis. • You should consider invoking the reason Alcoholism since it is a symptom unique to the correct diagnosis and present in the clinical case. • The symptom Contact dermatitis does not meet the criteria for a strong reason in this case. • The symptom Dry skin does not meet the criteria for a strong reason in this case. • The symptom Dermal atrophy does not meet the criteria for a strong reason in this case.
The Expert’s Explanation for this case attributes the patient’s condition to Porphyria Cutanea Tarda (PCT), citing factors such as the patient’s gender, alcohol abuse, and the presence of blistering on photoexposed skin. These align with our top-scoring reasons in the Assessment of Reasons, demonstrating the agreement between the expert’s explanation and our assessment. In the expert’s explanation, the symptom “Abnormal hair morphology” is not mentioned. However, our methodology identifies it as a significant reason that could enhance the explanation. This symptom is common in the incorrect disease Epidermolysis bullosa acquisita, but it is not present in the clinical case. Therefore, its absence provides a strong reason to discard Epidermolysis bullosa acquisita as the correct diagnosis. This additional information could potentially enhance the expert’s explanation by providing further evidence to support the correct diagnosis and rule out other alternatives. This demonstrates the capability of our approach not only to validate the reasons used by the expert but also to suggest new pieces of information that could enrich the explanation.
In this paper, we present a novel approach to recognizing the intricate cognitive processes that underpin the selection of explanations and strive to emulate these processes in a methodical and transparent manner. By incorporating the principles of abnormality and contrastive explanation, we ensure that our approach is attuned to the subtleties of explanation selection in real-world contexts, with a particular focus on the medical domain. Moreover, by leveraging the Human Phenotype Ontology and medical named entity recognition, we are able to identify and assess potential reasons in a systematic and data-driven manner. This not only allows us to assess the explanations that are consistent with the expert’s perspective but also to suggest additional pieces of information that could enhance the explanation itself.
Our work provides a significant step forward in the development of AI systems that can automatically assess explanations both in a cognitively plausible and contextually sensitive manner. It is driven by the necessity for a systematic and transparent approach to assessing the pertinence of reasons used in explanations, particularly in the medical domain. The deterministic nature of the prevalence function ensures the transparency of our approach.
A key advantage of our methodology is its modularity, with distinct components that can be modified or replaced to adapt the approach to new domains or tasks. The prevalence function encapsulates the reasoning model for scoring explanations, while the sentence matching technique, matching strategy, and knowledge base instantiate components tailored to assessing clinical rationales. Each of these modules could be swapped with alternatives better suited for a diferent domain or application. For example, the prevalence function could be re-designed to match commonsense rather than medical reasoning, while more general lexical resources like WordNet could replace domain-specific ontologies. This flexibility broadens the applicability of our techniques to assess explanations in a wide range of settings. Future work includes the application of this approach to education in medicine, by providing a systematic and transparent way of evaluating the reasons given in medical residents’ explanations, and to online discussions in medical fora by helping users and moderators to identify high-quality and low-quality explanations. Additionally, our current implementation is limited to evaluating explanations based solely on symptoms. However, physicians may consider other types of clinical information, such as lab findings, family history, or patient demographics. To address this, we plan to expand the scope of our approach by extracting and encoding additional clinical entities, such as test results, into ontology concepts like HPO. This will enable assessing explanations across a broader range of relevant clinical data.
This work has been supported by the French government, through the 3IA Côte d’Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR- 19-P3IA-0002. This work was supported by the CHIST-ERA grant of the Call XAI 2019 of the ANR with the grant number Project-ANR-21-CHR4-0002.