Machine learning (ML) models, demonstrably powerful, sufer from a lack of interpretability. The absence of transparency, often referred to as the black box nature of ML models, undermines trust and urges the need for eforts to enhance their explainability. Explainable AI (XAI) techniques address this challenge by providing frameworks and methods to explain the internal decision-making processes of these complex models. Techniques like Counterfactual Explanations (CF) and Feature Importance play a crucial role in achieving this goal. Furthermore, high-quality and diverse data remains the foundational element for robust and trustworthy ML applications. In many applications, the data used to train ML and XAI explainers contain sensitive information. In this context, numerous privacy-preserving techniques can be employed to safeguard sensitive information in the data, such as diferential privacy. Subsequently, a conflict between XAI and privacy solutions emerges due to their opposing goals. Since XAI techniques provide reasoning for the model behavior, they reveal information relative to ML models, such as their decision boundaries, the values of features, or the gradients of deep learning models when explanations are exposed to a third entity. Attackers can initiate privacy breaching attacks using these explanations, to perform model extraction, inference, and membership attacks. This dilemma underscores the challenge of finding the right equilibrium between understanding ML decision-making and safeguarding privacy.
In recent years, advancements in Artificial Intelligence (AI) have expanded beyond the primary
objective of predictive capabilities. Although accurate predictions are crucial, an equally
important goal has emerged: ensuring explainability. Explainability in Machine Learning (ML)
models has become a critical objective for making clear and justifiable predictions, especially in
high-stakes social decisions. It is essential for these models to ofer clear and comprehensible
reasons for their predictions and decisions [
In many applications, the data used to train AI and XAI models contain sensitive information
about individuals, such as medical records, or financial transactions, which the GDPR [
In order to enhance transparency, XAI techniques provide the necessary tools to open up
complex black boxes and shed light on how AI decisions are made [
Post-hoc explainability is a technique used to gain insight into the decision-making process of a trained ML model. In this context, post-hoc means that the model’s interpretability is addressed after its training, regardless of its complexity or the algorithms used. The approach primarily revolves around the act of querying the model with diverse sets of input data to observe how it reacts to diferent scenarios. Through these interactions, we can efectively map out the decision boundaries the model uses, shedding light on what factors influence its predictions.
Visualizations and explanations can then be applied to make these insights more accessible and human-friendly, ultimately enabling a better understanding of the model’s predictions. These visual aids are essential in making the insights gained more accessible to data scientists, end users, and domain experts who are willing to understand why the model is making specific predictions. By going through this process, post-hoc explainability serves a vital role in improving model transparency and building trust in its performance.
Understanding an AI system with XAI relies on its training data, process, and model. Therefore, XAI can be applied throughout the entire AI development pipeline. Specifically, it can be applied in diferent stages of modeling, such as before, during, and after (post-modeling explainability). In this work, the primary emphasis will be on post-modeling XAI (Post-hoc), since ML models are often developed with only predictive performance in mind.
Feature Importance Feature Importance (FI) explanations involve assigning a quantitative
measure in the form of a numerical score to each input feature within a given model. The
primary goal of calculating FI is to discern which features have influential efects on the model’s
predictions and which ones have a relatively lesser impact. These importance scores help
practitioners and data scientists gain insights into which factors are most critical in influencing
those decisions. Features that, when modified, cause more substantial shifts in the model’s
output are considered more important because they have a greater influence on the final
prediction. For deep learning models, many feature-based explanation functions are
gradientbased techniques that analyze the gradient flow through a model. Approaches such as Layer-wise
Relevance Propagation (LRP) [
Data protection and privacy is one of the primary dimensions in ML and AI. It involves ensuring that the data used to train and test ML models does not expose sensitive information about individuals or entities. This is particularly critical when dealing with datasets that contain personally identifiable information or confidential details. Techniques like anonymization and DP have emerged as valuable tools in the data privacy field. They allow us to protect the privacy of individuals represented in the data, even as we leverage it to train models. Beyond data privacy, model privacy is also a pressing concern. The architecture of ML models can be susceptible to privacy breaches. Models may unintentionally encode information about the training data they were exposed to, and this could pose risks when shared or deployed publicly on the cloud as MLaaS. Attacks such as model extraction, inversion, or membership inference can exploit these vulnerabilities (Details in the following sections and Fig. 1). However, privacy is not included in the default behavior of most ML algorithms. They tend to learn not just the general trends but also the specifics of the data, potentially revealing sensitive information when the model is made public. In an ideal scenario, we want these algorithms to focus on extracting general trends and patterns from the data while deliberately avoiding the inclusion of specific details about the data. This emphasis on distilling general patterns means that the algorithms should primarily capture the fundamental, common insights that are valuable for decision-making, aligning with privacy concerns, as they identify important details without risking individual privacy. XAI can inadvertently compromise privacy by revealing sensitive information about the model’s decision boundaries. Moreover, the process of returning real data points with CFs can inadvertently expose specific instances from the training set or behaviors. Also, the process of assigning FI scores exposes the values of gradients and the feature values themselves. This conflict makes striking the right balance between model explainability and data privacy crucial to ensuring that XAI enhances our understanding of AI systems without leaking individual privacy.
A membership inference attack (MIA) is a privacy-related threat in ML where an adversary
attempts to determine whether a specific data point was part of the training dataset of a deployed
model [
Model extraction (MEA) is a class of attacks where an adversary tries to reverse-engineer a
target model by observing its behavior and querying it. MEA can potentially lead to the theft of
intellectual property compromising proprietary models [
[
A model inversion attack (MINA) is a privacy-related threat in ML where an adversary attempts
to reconstruct sensitive or private information about individual data points from trained model
predictions. In other words, the MINA task is to predict the input data, that is, the original
dataset for the target model. In [
We pose the following research questions (RQs): 1. To what extent does the utilization of known privacy-preserving techniques, such as DP, efectively safeguard privacy and prevent information leakage when combined with explanations provided by XAI? 2. Can we produce high-quality XAI explanations while safeguarding privacy to mitigate potential vulnerabilities to attacks? 3. Which approach, privacy-preserving XAI or privacy-preserving ML, ofers a more efective solution for safeguarding sensitive information in XAI systems?
To address RQ1, we aim to evaluate the trade-of and assess the efectiveness of existing privacy-preserving techniques (e.g., DP) in mitigating information leakage when combined with XAI explanations for CFs and FI. This will involve investigating the extent to which explanations can be exploited for privacy attacks like MIA, MEA, or MENA.
To address RQ2, we aim to explore the possibility of generating high-fidelity XAI explanations while simultaneously safeguarding privacy.
Such approaches aim to develop an XAI framework that concurrently optimizes two objectives: i) generating high-quality CFS, and ii) adhering to pre-defined privacy constraints . Furthermore, the integration of DP during the backpropagation of gradients for FI computation is another promising avenue for investigation.
To address RQ3, we will conduct a comparative analysis of privacy-preserving XAI and privacy-preserving ML techniques. This analysis will evaluate their strengths and weaknesses in safeguarding sensitive information within XAI systems. By comprehensively assessing these aspects, we aim to identify the approach that ofers a more robust and enduring mechanism for privacy protection within XAI applications, covering diferent types of data.
In the initial research, I explored CF generation through RL, with the specific goal of constructing an explainer that operates independently of input data. The investigation then progressed to a more in-depth examination of CFs, focusing on their potential for information leakage and their ability to reveal the decision boundaries of ML models. To reach this aim, a new methodology is proposed to carry out MEA through a concept known as knowledge distillation (KD). I also delved into the domain of explainable deep learning methods within distributed systems, such as Vertical Split Learning (VSL), aiming to evaluate the potential information disclosure resulting from FI across various entities. In addition, I analyzed the impact of DP on the explainability of anomaly detection (AD) models. More specifically: 1. Explored how RL can be leveraged to generate CF explanations without relying on the dataset as input to the explainer. The main aim is to let the CF generator learn generalizable patterns from the training data without exposing it. The explainer determines which features to modify and by how much, by maximizing a custom reward function designed to jointly optimize various metrics. 2. Designed a new attack approach to evaluate the use of KD for an MEA in scenarios where CFs are given to an attacker. I benefit from the property of KD and the process of transferring knowledge from a large model to a smaller one. The findings reveal that employing KD with the presence of CFs can indeed yield successful MEA. 3. Proposed an approach to generate private CFs I introduce the concept of DP within the GANs CF generation pipeline to generate CFs that deviate from the statistical properties of the confidential dataset, ofering a layer of protection against potential privacy breaches. 4. Explored VSL strategies and performed experiments to explore the risk of information leakage regarding the original features using gradient-based explanations (IG and DeepLIFT). My application of VSL focused on a use case related to Network Function Virtualization. My findings highlight how an attacker on the server side can exploit XAI techniques to achieve additional tasks, without access to the original features. 5. Explored DP with AD Analyzed the trade-of between privacy achieved by DP and explainability achieved using SHAP.
This PhD research aims to achieve significant advancements in bridging the critical gap between XAI and data privacy. We will address the inherent conflict between providing users with clear explanations of AI models and protecting their sensitive data (privacy). We aim to develop a defense mechanism in the form of high-quality explanations while simultaneously ensuring privacy.