Algorithmic diagnosis using Electrocardiogram (ECG) signals for various cardiovascular diseases is an important step towards developing AI-assisted healthcare systems. Explaining the predictions of algorithmic decision through machine learning models seems to be absolutely necessary for practical purposes to inculcate trust and transparency. Shapley value-based additive feature importance explanation is supported with game theoretical axioms. In this paper, we demonstrate that the Shapley value-based features do indeed directly impact the model predictability under subtractive counterfactual setup. It is validated through adversarial machine learning condition as removal-based explanations that quantify the influence of each of the inputs through simulating input removal process. We show that the model's prediction capability degradation and the model hardening with adversarial training are coupled with Shapley value attributed important features as subtractive counterfactual reasoning. Specifically, we empirically confirm that the Shapley value attributed important features put the model under lesser stress under the evasion attack and the model hardening outcome becomes more robust. We substantiate our claim with empirical results, which are demonstrated on diverse ECG data of publicly available UCR time series dataset.
EXPLIMED - First Workshop on Explainable Artificial Intelligence for the medical domain - 19-20 October 2024, Santiago de
Compostela, Spain
* Corresponding author
sensor (mostly single-lead ECG sensor) as well as to provide explanatory basis towards the machine
learning prediction such that emergency care service can be deployed which can potentially lead to
reduced mortality rate and to avoid the clinical burden of delayed intervention. Further, we need
to ensure trust, data security and privacy of the smart healthcare eco-system [
In our context, AI-assistant is primarily a deep learning model that analyzes the single-lead ECG and
generates algorithmic prediction on the plausible CVD condition of the user along with alert generation,
when necessary. While accuracy of the CVD condition detection is important, the pure data-driven
approach is not suficient for the acceptance by the medical fraternity. An explanation of the results is
of utmost importance [
However, the model explanation’s validation is not studied well to confirm the eficacy of the SHAP method. In this paper, we consider ECG analysis as the exemplary application to demonstrate that the SHAP or Shapley value-based additive feature explanation provides consistent and intuitive explanation through adversarial machine learning set up under counterfactual robustness through explanation by removal [20]. We are further motivated by CXPlain [21] that removes single or a group of inputs to measure the function’s loss as the causal objective. Consequently, we follow subtractive counterfactualization. In additive counterfactuals, extra information is added to study the response, and in subtractive counterfactuals some of the given information is removed to study the response. From the understanding that additional information provisioning is an expensive exercise when medical data collection and annotation are concerned, we focus on subtractive counterfactuals. For empirical study, we experiment with diferent ECG data from UCR time series [ 22], which is the benchmark archive for time series classification problems, and we demonstrate empirical support for the consistent explanation capability of Shapley value-based explanation method. 1https://www.medicalindependent.ie/in-the-news/conference/ai-promises-the-gift-of-time-to-cardiologists/
ECG is a time series signal which is an ordered set of real values collected over time intervals and it is represented as: x = [1, 2, 3, ..., ], x ∈ R , where x consists of scalar measurements over a time period indexed by 1, 2, 3, ...., . Training ECG dataset X = [x(1), x(2), . . . , x(N)], where each of x(), = 1, 2, . . . , N consists of N number of ECG signals with corresponding labels of disease class Y and the complete training dataset is , where = [X , Y ]. Here we consider supervised learning problem to classify the given ECG signal into predicted disease class, where we construct a model ℎ (.), which is parameterized by describing the joint distribution (x,y) and we generate trained model .
The first problem is to build an accurate classification model from under diferent practical
challenges like less number of training examples, etc. Previously, it is demonstrated that sophisticated
deep neural network model like 2 is a suitable choice as the baseline deep neural network
model [
1. (Definition I) (Transferable utility game). A game that maps v: 2N → R such that v(∅) = 0 with the interpretation of v( ) where in 2N, as the estimated value of coalition and the value function v( ) finds out the collective payof for each of the player in the cooperation assumption. In our context, the model is trained with nℎ sample on all possible subset ⊆ 2N and we estimate for each of training samples. 2. (Definition II) (Marginal contribution). The marginal contribution ∆ v(n, ) of player n with respect to the coalition is defined as ∆ v(n, ) = v( ∪ n) − v( ).
We define Λ to be the integer permutations up to total number of given inputs (N) and ∈ Λ and the predecessor set of players preceding nℎ player in is represented as: , = { : () < ()}. Accordingly, Shapley value v(n) of nℎ player with the function v is: v(n) = N1! ∑︀ ∈Λ ∆ v(n, , ).
The Shapley value v(n) of nℎ is computed through permutation logic for each of the training sample
as [
The training samples with higher values of v(n) are the ones that contribute more to the learning of
the model [
Counterfactual setup is typically expressed as (|′, ′) which represents the probability that the outcome = is observed when the input is = under the actual observation of = ′ and = ′. A valid counterfactual is the one which is in fact classified as the desired class. Under a counterfactually robust classifier, the resistant to changes in is high and the classifier attempts to classify as , instead of ′, i.e. the distance between the actual observation and counterfactual observation ′ should be high for an adversarially robust model. The Shapley value-based explanation can simulate each of the input feature not being present in the distribution such that the prediction outcome is explained under the cooperative game scenario contrasting to the distribution change due to the absence of that input. Basically, Shapley value-based explanation is performed by asking a set of contrastive questions. Therefore, we encounter five types of data conditions.
• Training dataset which is the given labelled training set = [X , Y ] with X = [x(1), x(2), . . . , x(N)], and corresponding labels Y . • Test dataset is the given testing dataset that is independent from the training dataset. • Shapley value estimated feature attributed negative dataset − ℎ which consists of training set (− ℎ ⊂ ) discarding the top % of the positive (important) inputs ( can be 5, 10,15,20,.., practically is to be restricted <50). which is the simulated counter• Contrasting test inputs simulating counterfactuals
factual test inputs generated using (for e.g.) DeepFool algorithm [26]. • Model hardening training dataset ℎ which is the adversarial training input that provides the model (this model is trained with ) or ℳ (this model is trained with − ℎ) to train with augmentation through adversarial examples to counter the degradation of .
performance on
These datasets set the stage to understand the robustness of the Shapley value-based explanation under adversarial machine learning with subtractive counterfactual set up. These datasets condition the base model and ℳ. The outcomes in terms of test accuracy () deliver the required understanding of the capability and robustness of the Shapley value-based explanation when measured under counterfactuals. We consider model’s classification performance loss in the absence of an input or a subset of the inputs to compute the explanations as a function and the associated outcome as: : (x) → R. The value that is associated with the input x indicates the behavior of the model. The following outcomes provide us the quantified idea of the Shapley value-based explanation.
• Baseline test accuracy is , which is the test accuracy when the model is trained with and tested over and the corresponding trained model is . • Test accuracy with Shapley value estimated feature attributed negative dataset is − ℎ, which is the test accuracy when the model is trained with − ℎ and tested over and the corresponding trained model is ℳ. • Test accuracy over counterfactuals tested with is , which is the test accuracy of . when tested over • Test accuracy over counterfactuals tested with ℳ is , which is the test accuracy − ℎ .
of the model ℳ when tested over • Test accuracy over counterfactuals with trained with model hardening training dataset is
− ℎ, which is the test accuracy of the model hardened with adversarial training input ℎ and tested over
. • Test accuracy over counterfactuals with ℳ trained with model hardening training dataset is −ℎ− ℎ which is the test accuracy of the model ℳ hardened with adversarial training input ℎ and tested over
.
Our hypothesis of Shapley value-based explanation eficacy under counterfactuals is stemmed from the intuition that explanations are strongly related with the counterfactual explanations and adversarial robustness. The model which consists of all the training inputs including the high Shapley-valued or the important ones is superior not only over the given test inputs , but also over the contrasting than the model ℳ that is trained by discarding the test inputs simulating counterfactuals important inputs according to SHAP. Consequently, when the model gets hardened with augmented data ℎ, the response of should be similarly better than ℳ over counterfactual test inputs. More specifically, we need to establish that: 1. ≧ − ℎ, 2. ≧
− ℎ 3. −ℎ ≧ −ℎ− ℎ
Furthermore, consider 1 > 2, 1, 2 ∈ N+ and we denote − ℎ( 1), − ℎ( 2) as the top 1% and 2% Shapley valued inputs removed in the model training. Our second hypothesis is: 1. − ℎ( 2) ≧ − ℎ( 1), 2. − ℎ( 2) ≧
− ℎ( 1) 3. − ℎ− ℎ( 2) ≧
− ℎ− ℎ( 1)
While the above hypotheses are not proven, we establish our claim with empirical support over practical ECG datasets, given the model explainability is an important aspect of AI-assistive cardiac care that uses automated ECG analysis for diverse decision making and taking related actions.
In this study, we experiment with four ECG datasets, publicly available in the UCR archive, which is a benchmark dataset for timeseries classification 2 [27]. Dataset description is described in the Table 1. These datasets (Table 1) consist of separate training and testing parts. As per the general convention,
the classification performance or eficacy of the model is quantified by the test accuracy measure [ 28]). The training and test datasets are publicly available [27], which are standard timeseries classification benchmark data. Training-test partitioning are done by the database creator 3 .
The model is developed in Python 3.5.4 on Tensorflow 1.4.0 and Keras 2.1.2 libraries. The model is trained in two Nvidia GeForce GTX 1080 GPUs of 10 GB memory with 64-bit x86 architecture, 2.60GHz clock speed 16 core Intel Xeon E5-2623 v4 CPU. For SHAP implementation, we use DeepLIFT algorithm [29] through DeepExplainer implementation 4. To minimize the impact of non-reproducibility due to run-to-run variability owing to the non-determinism in typcal neural networks [30] 5, we experiment with more than 40 diferent random seeds and the reported empirical results are the highest occurring test accuracy values.
Firstly, we depict the performance of the deep neural network model for ECG classification. We design the ECG classification model following Residual Network (ResNet) architecture with novel restrained learning principle [23]. ResNet transforms the conventional layered representation learning and learns (x) = (x) + x at every layer of the network [24] so that the information in x gets a direct path to lfow into the network benefiting better learnability of the model. While ResNet provides substantial advantage, we intend to ensure that the learing of ECG signals to get better as ECG is also well-defined in the spectral domain. Thus, following the − 2 architecture [23], the deep neural network is formed with two parallel ResNet channels, where the ℎ1 1 learns as: 1(x) = (x) + (x) and ℎ2 2 learns as: 2(x) = (x) + x that ensures more detailed learning from the given ECG signals, where, (x) refers to the Fast Fourier Transform of input vector x. These two parallel ResNet channels are merged together to constitute the final block of representation (x), which is followed by a Global Average Pooling layer. Cross-entropy is the loss function and softmax function is the final classification output layer. The model architecture as two parallel ResNet channels is shown in Fig 2. The model hyperparameters are inspired from [23] and we have not performed any additional hyperparameter searching methods.
Since ECG classification is part of time series classification task, we consider the baseline and state-of-the-art algorithms that are well-studied in time series classification tasks. More importantly, we choose the state-of-the-art algorithms that use UCR [27] for their experimental study forexact comparison. The state-of-the-art comparison includes 1NN-DTW-based model [31], COTE [32],time series ResNet [33], TS-Chief [34], Proximity Forest (PF) [35] and Catch22 [36]. In Table 2, the comparative study of the test accuracies from the state-of-the-art algorithms and our proposed model are shown. We can positively conclude that the proposed model performs ECG classification efectively and it is in fact a state-of-the-art model.
Next, we perform the important study to understand the empirical support to form the basis of Shapley
value-based explainability through the lens of counterfactual set up with adversarial machine learning. From the hypothesis as stated in 3, we empirically study the impact of the Shapley explanation in a counter-intuitive approach and quantitatively evaluate the hypothesis through diferent test accuracies , − ℎ(10,15,20), , −ℎ(10,15,20), −ℎ, −ℎ(10,15,20)− ℎ, where, 1 = 5, 2 = 10, 2 = 15. The baseline test accuracy is , which is from the trained model that is trained with . We find the test accuracies from the models with Shapley value estimated feature attributed negative datasets at diferent levels of removal of the input training samples (removing top 10%, 15%, 20% of the input samples respectively) denoted as − ℎ(10,15,20) for the trained model ℳ10,15,20 when the model is trained with − ℎ(10,15,20). The empirical results are depicted in 3, 4, and 5. The most interesting part is the consistent degradation of the performance with more Shapley-negative training (higher amount of top input removal in the training set, i.e. with higher values) and the consistency is similarly observed in the counterfactual simulating contrasting testing as well as in case of model hardening. While performance degradation in Shapley-negative training indicates the impact of the Shapley value explained inputs towards the predictability of the model, the trend of lesser recovery of higher Shapley negative training (for e.g., 15%, 20%) demonstrates the support of explainability through Shapley attribution under subtractive counterfactual set up (− ℎ(10) ≥ − ℎ(15) ≥ − ℎ(20) for each of the experimental datasets as well as −ℎ(10)− ℎ ≥ −ℎ(15)− ℎ ≥ −ℎ(20)− ℎ for each of the experimental datasets) confirming our hypothesis. More precisely, we provide empirical evidence to support the explanation process of Shapley value-based input attribution which is strongly related to counterfactuals provisioning.
We intend to mention that the application space is restricted to ECG classification due to its immediate importance in the development and deployment of smart cardiovascular system. However, we can extend the application for other relevant healthcare data analysis and classification tasks given that the model-level explainability is an utmost importance property for the acceptance of AI-assistive solutions in healthcare domains including cardiovascular care for practical purpose.
Automation in the medical care, particularly in critical care that includes cardiovascular care not only increases the eficiency in the overall medical process, but also can lead to timely intervention through AI assistant that can potentially result in lesser mortality rate and reduced clinical burden. In this paper, we anchor upon smart cardiovascular system using ECG sensor for in-home, remote and emergency care that can enable emergency cardiovascular care without delaying the life-saving intervention process. However, to embrace the automation or algorithmic clinical condition detection and alert generation for initiating required cardiovascular care, the machine learning algorithm requires to provide modellevel explanation to justify the algorithmic prediction of the disease condition. Shapley value-based explanations backed by strong theoretical foundation from coalition game axioms is the apt choice and we formulate our hypothesis on the applicability of Shapley based explanations under subtractive counterfactual reasoning set up and demonstrated through empirical study on number of ECG datasets. However, we like to mention that the Shapley value computation is computationally challenging and DeepExplain with DeepLift algorithm or its variants can provide better computational eficiency. The method that we have proposed in this paper to support Shapley value-based explanations through subtractive counterfactual reasoning is generic in nature and we have chosen ResNet architecture due to its strong performance in diferent related classification tasks. A study with other architectures will certainly make the claim stronger. Further, in our future work, we intend to work on the theoretical basis of the proposed hypotheses and to provide more empirical evidences on related medical domains. We also intend to point out that the corresponds of the clinical explanation with Shapley value-based statistical explanation are not theoretically, hypothetically or empirically established. In this paper, our main motivation is to understand the influence of input training instances towards the model’s predictability, which in turn provides quantified explanation. Future research scope certainly includes the quantified explainability with qualitative explainability with respect to relevant and standard clinical domain knowledge.
We thank and acknowledge the support and continuous encouragement of Leandro Marin, University of Murcia, Spain for this research work.. [18] S. Katsushika, S. Kodera, S. Sawano, H. Shinohara, N. Setoguchi, K. Tanabe, Y. Higashikuni, N. Takeda, K. Fujiu, M. Daimon, et al., An explainable artificial intelligence-enabled electrocardiogram analysis model for the classification of reduced left ventricular function, European Heart Journal-Digital Health 4 (2023) 254–264. [19] W. Sun, S. V. Kalmady, N. Sepehrvand, A. Salimi, Y. Nademi, K. Bainey, J. A. Ezekowitz, R. Greiner, A. Hindle, F. A. McAlister, et al., Towards artificial intelligence-based learning health system for population-level mortality prediction using electrocardiograms, NPJ Digital Medicine 6 (2023) 21. [20] I. Covert, S. Lundberg, S.-I. Lee, Explaining by removing: A unified framework for model explanation, Journal of Machine Learning Research 22 (2021) 1–90. [21] P. Schwab, W. Karlen, Cxplain: Causal explanations for model interpretation under uncertainty,
Advances in neural information processing systems 32 (2019). [22] H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana,
E. Keogh, The ucr time series archive, IEEE/CAA Journal of Automatica Sinica 6 (2019) 1293–1305. [23] A. Ukil, A. J. Jara, L. Marin, Blend-res 2 net: Blended representation space by transformation of residual mapping with restrained learning for time series classification, in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 3555–3559. [24] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. [25] A. E. Roth, The Shapley value: essays in honor of Lloyd S. Shapley, Cambridge University Press, 1988. [26] S.-M. Moosavi-Dezfooli, A. Fawzi, P. Frossard, Deepfool: a simple and accurate method to fool deep neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2574–2582. [27] H. A. Dau, E. Keogh, K. Kamgar, C.-C. M. Yeh, Y. Zhu, S. Gharghabi, C. A. Ratanamahatana, Yanping, B. Hu, N. Begum, A. Bagnall, A. Mueen, G. Batista, Hexagon-ML, The ucr time series classification archive, 2018. [28] A. Bagnall, J. Lines, A. Bostrom, J. Large, E. Keogh, The great time series classification bake of: a review and experimental evaluation of recent algorithmic advances, Data mining and knowledge discovery 31 (2017) 606–660. [29] A. Shrikumar, P. Greenside, A. Kundaje, Learning important features through propagating activation diferences, in: International conference on machine learning, PMLR, 2017, pp. 3145–3153. [30] D. Zhuang, X. Zhang, S. Song, S. Hooker, Randomness in neural network training: Characterizing the impact of tooling, Proceedings of Machine Learning and Systems 4 (2022) 316–336. [31] J. Lines, A. Bagnall, Time series classification with ensembles of elastic distance measures, Data
Mining and Knowledge Discovery 29 (2015) 565–592. [32] A. Bagnall, J. Lines, J. Hills, A. Bostrom, Time-series classification with cote: the collective of transformation-based ensembles, IEEE Transactions on Knowledge and Data Engineering 27 (2015) 2522–2535. [33] Z. Wang, W. Yan, T. Oates, Time series classification from scratch with deep neural networks: A strong baseline, in: 2017 International joint conference on neural networks (IJCNN), IEEE, 2017, pp. 1578–1585. [34] A. Shifaz, C. Pelletier, F. Petitjean, G. I. Webb, Ts-chief: a scalable and accurate forest algorithm for time series classification, Data Mining and Knowledge Discovery 34 (2020) 742–775. [35] B. Lucas, A. Shifaz, C. Pelletier, L. O’Neill, N. Zaidi, B. Goethals, F. Petitjean, G. I. Webb, Proximity forest: an efective and scalable distance-based classifier for time series, Data Mining and Knowledge Discovery 33 (2019) 607–635. [36] C. H. Lubba, S. S. Sethi, P. Knaute, S. R. Schultz, B. D. Fulcher, N. S. Jones, catch22: Canonical time-series characteristics, Data Mining and Knowledge Discovery 33 (2019) 1821–1852.