It is generally accepted that Machine Learning and AI models can be made “trustworthy” by providing explanations that justify their predictions, and multiple techniques have been proposed to achieve this. These are collectively referred to as model-based explanations, and the kind of models that can be explained using those are referred to as XAI. There is, however, also a complementary need to provide data explanations that account for the dataset selection and data engineering steps that precede model training. Data explanations can be obtained by capturing and then examining the provenance of the data in a training set. In this position paper, we suggest that there is value in combining these two types of explanations, producing an “eXplainable End-to-End” (XEE) backward path that originates from model predictions and traces all the way back to raw data, using a combination of XAI and provenance methods. We demonstrate how this can be achieved in practice using Influence Functions, a well-known XAI technique, in combination with PROLIT, a provenance-based tool that we have been developing to provide data explanations. We provide a simple but representative example and suggest the next steps for more in-depth research into XEE.
Multiple methods are available to achieve transparency and explainability of predictions made
by Machine Learning and AI models [
In this paper, we explore and illustrate the idea of combining model explanations with data explanations to provide an end-to-end account of a model’s prediction. More precisely, consider a training set that was constructed from a collection of initial datasets that are processed using a workflow consisting of data transformation operators.
The data explanation of data point in the training set is ’s provenance, for which we adopt the W3C PROV model (http://www.w3.org/TR/prov-dm/) and its standard graph representation
Such end-to-end explanations are useful in many practical settings. For instance, a data preparation pipeline may include data transformations that may afect model performance by accidentally injecting out-of-distribution data points or by altering the set through data augmentation or other forms of synthetic data injection. In such cases, it is important to identify the influential data points and trace them back through the pipeline to understand where they come from.
Given a model ℎ : R → R trained on , model explanations and data explanations can be described in abstract terms using two functions. The first is a function that maps the outcome ˆ = ℎ(x) for input x to a set of influential training data points in the training set : ℐ(ˆ) = {1 . . . }, ∈
Conceptually, combining model and data explanations involves composing these two functions to return a unified explanation as a set of provenance graphs. We denote such composition as XEE ℎ (eXplanations End to End relative to model ℎ):
XEE ℎ(x) = {prov ()| ∈ ℐ(ℎ(x))}
The main contribution of this paper is a demonstration of how such composition may be
realised in a practical setting and a discussion of how the idea can be extended to more general
settings. Specifically: (i) we have adapted the Tracin algorithm [
The rest of the paper is organized as follows. In Section 2 we provide an overview of model explanations based on influence functions. In Section 3 we discuss data explanation describing PROLIT, a system we have developed for this purpose. In Section 4 we illustrate how model and data explanations can be combined and finally, in Section 5 we draw some conclusions and discuss future research on XEE.
Research into methods that make “black box models” interpretable is not new, with notable
model-agnostic algorithms like LIME (Local Interpretable Model-Agnostic Explanations) dating
back from 2016 [
Similarly, SHapley Additive exPlanations (SHAP) uses concepts from cooperative game theory
to assign Shapley values to features, quantifying their contribution to a model’s predictions.
Unlike LIME, SHAP provides both local and global explanations. It is particularly efective for
tree-based models and neural networks but can be computationally expensive for large datasets
or highly complex models [
The idea of using Influence Functions to support black-box explanations originates around
the same time [
A complete formalisation of the problem is beyond the scope of this description. Intuitively, the idea is to estimate the change in the parameters ∈ Θ of the model due to removing a single point from the training set. As we know, the training process calculates the optimal values for : 1 ∑︁ (, ) ˆ= arg min ∈Θ =1 where (, ) is the loss function and 1 . . . are the training points.
The change in parameters can be expressed as ˆ− − ,ˆwhere ˆ 1 ∑︁ (, ). − = arg min∈Θ ̸=
Influence functions remove the need to repeatedly retrain the model for each in order to calculate ˆ− . The idea is to slightly upweight in the loss function by a small perturbation , giving new parameters: ˆ 1 ∑︁ (, ) + (, ).
, = arg min∈Θ =1
Results from influence functions theory show that this can be achieved by calculating the Hessian
of (, ) , which does not require retraining but is itself computationally expensive. However,
the computational challenge of this calculation can be addressed using known approximation
techniques, making the method usable in practice (see [
These findings paved the way for a number of variants of the methods, for instance [
The provenance of tabular data that flows through a data science pipeline in preparation for use by machine learning algorithms is a graph representation of the sequence of transformations that alter both their schema and values. The PROLIT system [14] is designed to collect, store, manage, and query provenance information such as data flows through data preparation pipelines. It also leverages a Large Language Model (LLM) to allow users to query the provenance with questions expressed in natural language and to return short provenance narratives describing in words the operations performed on the data.
PROLIT relies on a provenance model that extends PROV [15] to capture provenance at diferent levels of granularity within a table, including atomic value, row, column, and whole table. This model, illustrated in Figure 1, represents provenance data with a graph structure involving Entity nodes (representing data values), Column nodes (representing entire columns of a dataset), and Activity nodes (representing operations on the data). Relationships between these nodes capture actions like data consumption (used), creation (GeneratedBy), removal (InvalidatedBy), and derivation (DerivedFrom) during data transformations. The belongsTo relationship links individual cells to their respective columns whereas the next relationship explicitly sequences activities.
PROLIT relies on a LLM in two ways. Firstly, it recognises individual operations performed on the data from an arbitrary Pandas program and creates a structured representation of their transformations, rewriting user-defined pipelines into a standardised format. This code segmentation allows PROLIT to precisely associate provenance to each code snippet. Secondly, it generates human-readable narratives of both the code snippets and of elements of the provenance graph in response to a user question.
More in detail, PROLIT takes a Python script that manipulates Pandas dataframes as input. It first rewrites the script, using a LLM to identify and document the activities. The PROLIT 1https://deel-ai.github.io/influenciae/ provenance generator then observes the execution of the rewritten script, producing a provenance graph fragment that captures input/output data dependencies for every activity. The complete provenance graph at the end of execution is written into a graph database (Neo4j), where it can be analysed using either native Neo4j access or through a LLM-based component that translates natural language questions into Cypher queries and then renders the resulting graph as a natural language narrative.
An example of provenance captured by PROLIT for a pipeline that first eliminates from a dataset the Purchased column, which acts as the target of a classification model, and then imputes null values occurring in the Age column is reported in Figure 2.
PROLIT addresses the problem of the excessive volume of provenance data that can be generated when tracking individual elements of a large dataset by enabling customization of the level of granularity to which the provenance is collected and queried. In particular, it can provide a sketch of the collected provenance, involving a few instances for each operation in the pipeline, and a view of the provenance at schema-level, showing only operations and involved columns, hiding their efect on individual data.
We now illustrate how model and data explanations can be combined, with a simple example based using the well-known Titanic tabular dataset, here denoted simply . Firstly we describe and the script used to prepare for model training. The model itself is a simple Multilayer Perceptron and is not shown in the detail. Then we perform a model prediction for a random test data point, and show the ranked list of influential records used in the prediction produced by Tracin. Finally, we query PROLIT with the top-ranking such record and display its provenance.
More in detail:
1. is processed using script . This results in training set ′ = (), derived from
through the operators in ;
2. PROLIT is used to collect the provenance graph ProvDS ′ () of each element in ′ by
observing the processing;
3. ′ is split into training and test sets ′, ′ and ′ is used to train a simple
neural network, producing a logistic regression classifier ℎ where ˆ = ℎ() ∈ {0, 1}
predicts whether or not a Titanic passengers survives.
4. The Tracin library, which implements the XAI approach described in [
As discussed above, each of the provenance records is rendered using natural language narratives.
The script , reported in Figure 4, operates on by performing three operations. 1. Drops unnecessary columns such as “Name”, “Ticket”, and “Cabin” which do not yield any useful information for the logistic regression model. 2. Imputes the mode for the “Embarked” column and the median for the “Age” column to handle missing values. 3. Performs one-hot encoding on categorical features “Pclass”, “Sex”, and “Embarked”.
This pipeline prepares the data to allow us to train our model, and as we are tracking
provenance, we are able to investigate the data before it has been altered. Tracin runs alongside
the training process as described in [
We should emphasise that validating influence, that is, understanding why these records have been ranked as highly influential, is an interesting question that is, however, beyond the scope of this example. In general, it would be interesting to study consistency across diferent influence-based methods, which are themeselves non-deterministic. In this example, one may speculate that the three survivors are “similar” to passenger 436 based on either their age, or sex.
We can now use PROLIT to retrieve the provenance of those passenger records, which will have been recorded when is executed, focusing on one of the influential passengers (ID 234). PROLIT records the provenance of a record or single attribute, only when changes occur anywhere in the record. In this example, the state of the record for passenger 234 changes twice, firstly due to imputation, and then due to the one-hot encoding. These state changes are reflected in the provenance graph for this record, as shown in Fig.6. The top part of the figure shows the state of the record before and after each of the changes, while the bottom part shows the corresponding provenance derivations (read from the right “back” to the past on the left). Note that no other entities appear in the provenance record, because no other attribute values have changed.
In this paper, we have argued for the importance of combining model-based explanations (XAI) with data explanations to create an eXplainable End-to-End (XEE) framework. By integrating techniques such as Influence Functions for model interpretability and a system that collects data provenance for pre-processing pipelines, we have demonstrated the feasibility of tracing predictions back to their raw data origins. This approach not only enhances the transparency Record state Record Provenance
234 Age:NaN Sex: f … Age: NaN usage
gen
Fillna() { arg: median(age) } 234 Age:5 Sex: f … Age:5 Sex: F usage
Encode gen Age:5 Sex_m:0
… Age:5 Sex_f: 1 Sex_m: 0 of machine learning models but also provides an improved understanding of the entire data-toprediction pipeline, fostering a better explainability of AI systems.
We have presented a practical example of how “black box” explanations of model inference, obtained using Influence Functions, can be efectively combined with data explanations obtained by capturing and then querying the provenance of training data points through a pre-processing pipeline. The supporting example relies on the assumption that model explanations consist of training data points. This assumption is necessary so that the two explanations can be combined, however, it rules out alternative model explanation methods like SHAP, as this returns a ranking of attributes in order of relevance, for the predicted data point itself. This assumption is not too strict, given the demonstrated applicability of Influence Functions, however further insight into model explainability may be required to properly generalise the method and ensure that provenance can be safely composed with XAI methods, leading to full XEE.
There are indeed several possible directions of future research aimed at developing the XEE framework, including the following: (i) the current idea relies on Influence Functions and PROLIT, which may face scalability challenges with large datasets or complex models — future work could explore more eficient algorithms or approximations to make XEE applicable to real-world, large-scale systems; (ii) while Influence Functions are efective, they represent only one of many XAI methods — investigating how other XAI techniques, such as SHAP, LIME, or counterfactual explanations, can be integrated into the XEE framework would broaden its applicability and robustness; (iii) the efectiveness of XEE should be evaluated not only from a technical perspective but also in terms of its usability and interpretability by end-users, including domain experts and non-technical stakeholders — user studies could provide insights into how XEE explanations can be exploited in practice and tailored to diferent audiences; (iv) the current framework focuses on structured, tabular data — extending XEE to handle unstructured data, such as text, images, or time-series data, would expand its scope and utility.
39th International Conference on Machine Learning, PMLR, 2022, pp. 13468–13504. URL: https://proceedings.mlr.press/v162/lin22h.html, iSSN: 2640-3498. [12] D. Liu, P. Cheng, H. Zhu, X. Tang, Y. Chen, X. Wang, W. Pan, Z. Ming, X. He, Diwift: Discovering instance-wise influential features for tabular data, in: Proceedings of the ACM Web Conference 2023, WWW ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 1673–1682. URL: https://doi.org/10.1145/3543507.3583382. doi:10.1145/ 3543507.3583382. [13] A. Picard, L. Hervier, T. Fel, D. Vigouroux, Influenciae: A library for tracing the influence back to the data-points, 2024. [14] P. L. Lazzaro, M. Lazzaro, P. Missier, R. Torlone, PROLIT: Supporting the Transparency of Data Preparation Pipelines through Narratives over Data Provenance, in: Proceedings of the 28th International Conference on Extending Database Technology (EDBT), OpenProceedings.org, Barcelona, Spain, 2025. [15] P. Missier, K. Belhajjame, J. Cheney, The W3C PROV family of specifications for modelling provenance metadata, in: Procs. EDBT’13 (Tutorial), ACM, Genova, Italy, 2013. URL: http://www.edbt.org/Proceedings/2013-Genova/papers/edbt/a80-missier.pdf.