Artificial Intelligence (AI) systems are increasingly relied upon in high-stakes domains such as healthcare, finance, and autonomous driving, as well as in high-value commercial applications like luxury automotive design and exclusive financial services, where decision-making must be both accurate and trustworthy. However, the opaque nature of many AI models raises concerns about transparency and accountability, driving the development of Explainable AI (XAI) techniques to foster trust. While these methods aim to improve interpretability, questions persist regarding the reliability and certainty of these explanations, particularly under varying conditions and sources of uncertainty. This underscores the need for robust trust measures to assess the validity and consistency of AI-generated explanations across diferent contexts. Consequently, the question shifts from "Can I trust this model?" to "To what extent can I trust the explanations and the reasoning behind the model's decisions?"-emphasizing the importance of reliable frameworks for explainability. To reliably quantify uncertainty in AI-generated predictions, we integrate conformal prediction, a distribution-free, model-agnostic framework that constructs prediction sets with statistically valid coverage guarantees, ensuring that the true outcome is included with a userspecified probability. By adapting to diferent tasks and data distributions, conformal prediction provides a robust foundation for uncertainty measurement and enables the generation of consistent, uncertainty-aware explanations across varying conditions. We term this approach “uncertainty-aware explanations”, providing systematic methods to assess the trustworthiness of AI insights in diverse contexts, including time series forecasting, classification, and other data-driven tasks. By addressing the relationship between uncertainty and explainability, this work aims to enhance the reliability of AI-driven decision-making in high-stakes environments.
As Artificial Intelligence (AI) systems become increasingly complex, explaining their decisions becomes
more challenging, leading to concerns about trust and reliability. This has prompted the development
of Explainable AI (XAI) to enhance transparency by providing human-interpretable explanations for
AI-driven decisions. XAI methods produce diferent types of explanations depending on the task,
afecting their applicability across domains. For example, a practitioner analyzing ECG signals to assess
a patient’s risk of developing cardiac disease requires retrospective analysis to identify key time intervals
contributing to the prediction (time series forecasting) and to rank other influential factors such as diet,
weight, and physical activity (time series classification). Despite advancements in Explainable AI (XAI),
there is a lack of robust methods to quantify the uncertainty in AI-generated explanations, leading
to potential overreliance on explanations provided in critical domains. As a result, a critical question
arises: Can we trust these explanations, and if so, to what extent? This highlights the need for rigorous
evaluation frameworks to assess the reliability, consistency, and validity of AI-generated explanations
across various contexts [
Uncertainty quantification (UQ) plays a critical role in AI systems, as it provides a measure of
confidence in the model’s predictions, helping practitioners assess the reliability of outputs, especially
in high-stakes applications [
Uncertainty-aware explanations in XAI enhance interpretability by revealing both the reasoning behind a model’s outputs and the confidence in those outputs. In this work, they are defined as explanations that capture how variations in uncertainty influence the trustworthiness of predictions. For example, in financial risk assessment, such an explanation might show that a loan applicant’s risk score is less reliable due to missing income data, with uncertainty rising by 30% compared to complete cases. This approach supports practitioners in evaluating the robustness of AI insights and making informed decisions under uncertainty.
The lack of uncertainty-aware explanations in AI can cause major issues, especially in high-stakes
areas like healthcare and autonomous driving [
Post-hoc explainability refers to techniques applied after a machine learning model has been trained,
aiming to interpret its predictions without altering the underlying model structure [
Model-agnostic explainability, a subset of post-hoc methods, is designed to be applicable to any
machine learning model, regardless of its architecture or complexity [
The growing demand for explainability stems from regulatory requirements, such as the European
Union’s General Data Protection Regulation (GDPR), which emphasizes the right to explanation [
Uncertainty quantification (UQ) is a fundamental aspect of machine learning that focuses on measuring
and interpreting the uncertainty associated with model predictions. This is particularly critical in
highstakes applications such as healthcare, autonomous systems, and financial forecasting, where decisions
based on overconfident predictions can lead to severe consequences [
Conformal prediction is a model-agnostic, non-parametric framework for uncertainty quantification (UQ) that provides valid confidence intervals without strong distributional assumptions [ 16]. Relying on the weaker exchangeability assumption rather than i.i.d., it calibrates prediction sets or intervals using a hold-out validation set to guarantee user-specified coverage (e.g., 95%) [ 17]. It applies to any model, including black-box architectures, and has recently been extended to time-series forecasting and high-dimensional data [18].
Let = {(, )}=1 denote a dataset, where represents the input features and represents the corresponding true label or value. Conformal prediction works as follows: 1. Nonconformity Measure: A nonconformity measure (, ) quantifies how unusual a pair (, ) is with respect to the model’s predictions. For example, in regression, (, ) could be the absolute residual | − ˆ|, where ˆ is the model’s prediction. 2. Calibration Set: A hold-out calibration set cal = {(, )}=1 is used to compute nonconformity scores = (, ) for each point in the calibration set. 3. Prediction Interval Construction: For a new input new, the conformal prediction framework constructs a prediction interval (new) such that:
(new) = { : (new, ) ≤ }, where is the (1 − )-th quantile of the nonconformity scores {}=1. This ensures that the interval (new) covers the true label new with probability 1 − ( is user-specified error rate).
1. Coverage: Coverage measures the proportion of true labels that fall within the prediction intervals or sets. For a dataset , the empirical coverage is defined as:
Coverage = 1 ∑︁ I( ∈ ()), =1 (1) where I(· ) is the indicator function. A well-calibrated conformal prediction framework ensures that the empirical coverage is approximately 1 − . 2. Set Size: Set size measures the size of the prediction sets or intervals. For classification tasks, it is the number of labels in the prediction set, and for regression tasks, it is the width of the prediction interval. Smaller set sizes indicate more precise predictions, while larger set sizes reflect higher uncertainty.
Integrating UQ and conformal prediction into machine learning pipelines improves decision-making by ofering insights into prediction reliability, crucial in safety-critical applications where overconfidence can lead to catastrophic outcomes [19]. In healthcare, conformal prediction provides condfience intervals for patient outcomes, aiding clinicians in making informed decisions. In autonomous systems, UQ assesses prediction reliability in dynamic environments.
These methods align with the growing focus on transparency and robustness in AI, supported by regulatory frameworks and industry standards [20]. Conformal prediction ofers valid coverage guarantees without distribution assumptions, is model-agnostic, and is applicable to various tasks, including time-series forecasting and high-dimensional data, with recent extensions like split and adaptive conformal prediction [18].
Related work linking explainability and conformal prediction remains limited. [21] propose CONFIDERAI, refining rule-based classifiers by combining conformal prediction with explainable ML for improved reliability. [22] introduces CONFINE, a framework for interpretable neural networks with robust uncertainty estimates. [23] explores oracle coaching to generate valid, eficient conformal classiifers optimized for specific test sets. [ 24] compares frequentist, Bayesian, and conformal uncertainty estimation, highlighting conformal methods for trustworthy confidence sets in model explanations. [25] apply XAI to cardiovascular risk prediction in COPD patients, comparing counterfactual methods and proposing counterfactual conformity for validation. [26] presents a conformal prediction-based framework for interpreting unsupervised node representations in graphs. Most relevant to this PhD is Calibrated Explanations (CE) [27], which provides stable, model-agnostic local feature importance maps with uncertainty quantification via Venn-Abers predictors [ 28]. In contrast, this work uses a perturbation-based, post-hoc, model-agnostic approach with classical conformal prediction, tailored to specific tasks and analyzing how predictive uncertainty shifts under varying calibration sets and systematic noise.
Following research questions (RQ) have been proposed for this research: 1. How can uncertainty in AI-predictions be efectively quantified to enhance trust and reliability in decision-making? 2. Which evaluation measures are needed to assess the validity/performance of conformal prediction, and how can we leverage them to produce uncertainty-aware explanations? 3. How do uncertainty-aware explanations generated by our proposed frameworks enhance decisionmaking and compare to conventional explainers across diverse real-world scenarios? To address RQ1, we begin by quantifying uncertainty in AI predictions using the flexible framework of conformal prediction. In classification tasks, this involves prediction sets with varying confidence levels , while in regression, it involves prediction intervals around outputs. Additionally, scalar uncertainty measures—such as variance from ensembles, dropout, input perturbations, or adversarial modifications—provide adaptable, task-specific metrics [ 29]. This phase identifies the most suitable uncertainty quantification methods across diferent tasks.
For RQ2, we extend conformal prediction to evaluate and communicate uncertainty in AI-generated explanations across tasks like classification , time series forecasting, and clustering. A key property of conformal prediction is that its coverage, while guaranteed on average (Equation 1), varies with calibration sets. We will explore how perturbing training or calibration data afects uncertainty and model performance, aiming to integrate these efects into reliable and transparent explanation frameworks.
Uncertainty metrics will be task-specific: in classification, we analyze prediction set size and coverage (Section 2.2.2); in forecasting, we assess changes in confidence interval bounds (Equation 3). Perturbing input features helps recalibrate models and track shifts in uncertainty. We also explore both local and global explainability.
In RQ3, we compare our uncertainty-aware explanation frameworks with SHAP, LIME, Saliency Maps, and Integrated Gradients. Efectiveness is tested via ablation of the top-ranked feature or segment; robustness by varying conformal prediction confidence levels; and faithfulness by comparison with inherently explainable models. These evaluations integrate uncertainty to enhance transparency and reliability. This research aims to develop task-specific uncertainty-aware explanations within a conformal prediction framework. Using a modular approach for classification, forecasting, and other tasks, it systematically measures and explains feature or segment contributions to predictive uncertainty.
For each task (e.g., classification, regression, time-series forecasting), the conformal prediction framework is tailored to produce prediction sets or intervals that quantify uncertainty, ensuring task-specific validity and interpretability [17].
Given an input and error rate ∈ (0, 1), the prediction set is: (2) (3) () = { ∈ | (, ) ≥ }, where is the label set, (, ) is a conformity score, and is a threshold ensuring coverage of 1 − . Regression Tasks For a predicted value ˆ(), the prediction interval is:
() = [ˆ() − , ˆ() + ], ensuring the interval captures the true value with probability 1 − .
We assess feature or segment contributions to uncertainty by applying systematic perturbations and measuring changes in Coverage and Set-size (Section 2.2.2). For classification , individual features are perturbed and models recalibrated; for forecasting, PELT segmentation [30] is used, and perturbing each segment reveals its impact on interval bounds (Equation 3). Comparisons with the unmodified baseline identify the main sources of predictive uncertainty.
The rationale for using conformal prediction for producing uncertainty-aware explanations is its ability to provide reliable uncertainty estimates independent of data distribution. However, despite its robust uncertainty quantification, conformal prediction often lacks systematic methods for explaining the sources of uncertainty and adapting to various tasks.
We hypothesize that perturbing features or segments and examining their efect on uncertainty metrics can yield meaningful, interpretable uncertainty-aware explanations. This is validated on diverse datasets to ensure robustness and generalizability. Efectiveness is assessed through ablation studies, where significant features or intervals are removed and performance is remeasured. We further evaluate robustness by varying confidence levels (1 − ), as defined in Section 5.1, and faithfulness by comparing our results to explanations from intrinsically interpretable models, verifying that they reflect genuine uncertainty sources rather than artifacts of the method.
In our research, we developed the global explainability framework ConformaSight [
We contributed to Fast Calibrated Explanations (FCE) [
In summary, the contributions up to date are as follows: 1. Leveraged conformal prediction to generate uncertainty-aware explanations for tabular data classification (ConformaSight): We designed a framework to identify which features contribute most to predictive uncertainty when subjected to significant perturbations, potentially causing the model to make incorrect predictions. The framework shows how calibration set perturbations influence prediction set outcomes, highlighting their impact on model performance. 2. Contributed FCE for rapid uncertainty-aware explanations for tabular data classification and regression: We proposed a method designed for generating faster, uncertainty-aware explanations by incorporating perturbation techniques from ConformaSight into the core elements of CE. This method boosts computational eficiency for real-time use while preserving uncertainty quantification in classification and probabilistic regression. 3. Extended conformal prediction to generate uncertainty-aware explanations for timeseries forecasting (ConformaSegment): We adapted our framework to time-series forecasting tasks, focusing on identifying the most critical time segments that contribute to predictive uncertainty, thereby influencing the accuracy of the forecasted values.
This PhD research aims to enhance trust in model decision-making by identifying key factors driving significant changes in model uncertainty. We explore how conformal prediction, which provides statistically guaranteed prediction sets with user-specified coverage, can be leveraged to generate uncertainty-aware explanations. Our goal is to develop a family of post-hoc, model-agnostic frameworks designed to produce reliable and interpretable explanations while advancing the transparency of conformal prediction-based explainability methods. Next, we aim to extend these frameworks to anomaly detection, synthetic data generation, and clustering, advancing transparent and generalizable uncertainty-aware explainability.
Fatima Rabia is a PhD student at DISI, University of Bologna, funded by PNRR (n. 9990) and Automobili Lamborghini S.p.A., Italy.
The author has used ChatGPT-4o exclusively for grammar checking and rephrasing; the author originally produced all content.