The emergence of deep learning networks raises a need for explainable AI so that users and domain experts can be confident applying them to high-risk decisions. In this paper, we leverage data from the latent space induced by deep learning models to learn stereotypical representations or “prototypes” during training to elucidate the algorithmic decision-making process. We study how leveraging prototypes effect classification decisions of two dimensional time-series data in a few different settings: (1) electrocardiogram (ECG) waveforms to detect clinical bradycardia, a slowing of heart rate, in preterm infants, (2) respiration waveforms to detect apnea of prematurity, and (3) audio waveforms to classify spoken digits. We improve upon existing models by optimizing for increased prototype diversity and robustness, visualize how these prototypes in the latent space are used by the model to distinguish classes, and show that prototypes are capable of learning features on two dimensional time-series data to produce explainable insights during classification tasks. We show that the prototypes are capable of learning real-world features - bradycardia in ECG, apnea in respiration, and articulation in speech - as well as features within sub-classes. Our novel work leverages learned prototypical framework on two dimensional time-series data to produce explainable insights during classification tasks. Equal Contribution
Despite the recent surge of machine learning, adoption of deep learning models in decision critical domains, such as healthcare, has been slow because of limited transparency and explanations in black-box algorithms. This observation points to the critical need for black-box models to offer interpretable, faithful explanations of their decisions so that practitioners in high-risk domains can trust model outputs and leverage their results. One such high-risk domain is treating preterm infants ( 10% of births worldwide) in the neonatal intensive care unit (NICU).
A common disorder observed in majority of preterm infants is recurrent episodes of apnea (cessation of breathing) and bradycardia (slowing of heart rate). Both of these spontaneous events may cause end organ damage related to hypoxemia (low oxygenation of blood) and ischemia (reduced blood flow) [Martin and Wilson, 2012]. Early detection of apnea and bradycardia can help prevent hypoxic-ischemic injury in tissue with high-metabolic demands [Schmid et al., 2015; Pichler et al., 2003] and prevent the cascade into intermittent hypoxia, which leads to complications of retinopathy, developmental delays, and neuropsychiatric disorders [Williamson et al., 2013; Poets et al., 2015; Di Fiore et al., 2015]. Leveraging explainability in deep neural network classification of these time series can reveal complex morphological and physiological features that clinicians may not readily see. Thus, machine learning algorithms need transparency in their decision-making process to highlight subtle patterns. One such technique in deep explainability is prototypes, a case-based reasoning technique.
Prototypes are representative examples, learned in-process during model training, that describe influential data regions in latent representations and provide insight into aggregated features across training data that are utilized by the model for classification. In contrast to post-hoc explainability, which trains a secondary model to infer decision reasoning from a primary model by only leveraging inputs and outputs, inprocess explainable methods offer faithful explanations of a primary model’s decisions [Rudin, 2018]. So, users who employ prototypes can confidently gain direct insight into the decisions algorithms are making for classification tasks.
On data with unclear class boundaries, in-process methods can misbehave. For example when the model in [Li et al., 2017] is applied to the MNIST dataset, the prototypes easily separate in the latent space because the latent data representation is separable and well-structured (Fig 1). However, when class boundaries and features do not form distinguishable clusters, learned prototypes become archetypes (extreme corner cases) that exist near the convex hull of the data in the latent space (Fig. 4). This phenomenon yields prototypes that represent extreme class types (i.e. archetypes) and can underperform on classifying data in overlapping class regions.
In this work, we provide a deep classification method with explainable insights for health time-series data. We introduce a prototype diversity penalty that explicitly accounts for prototype clustering and encourages the model to learn more diverse prototypes. These diverse prototypes will help focus on areas of the latent space where class separation is most difficult and least defined to improve classification accuracies. We show the utility of this approach on three tasks in two-dimensional time-series classification: (1) bradycardia from ECG; (2) apnea from respiration; and (3) spoken digits from audio waveforms. The two-dimensional representation of time-series provides an interpretable method for domain experts (e.g. clinicians) to understand the evolution of clinically relevant features based on visible phenotypes in timeseries data. Our work enables a closed-loop collaboration between experts and machine learning algorithms to accelerate the efficacy of outcome predictions. The learning algorithms can find nuance features through development of explainable prototypes, and the experts can fine-tune the algorithms by providing feedback through the regularization of the diversity penalty. This is especially important for clinician experts who need explainability in black-box models to understand and diagnose different pathological mechanisms. To the best of our knowledge this is the first application of prototypes and latent space analysis for health time-series data that could help reveal clinically relevant and explainable phenotypes to improve the baseline for standard of care with automatic monitoring and detection. 1.1
Explainable methods [Ribeiro et al., 2016; Caruana et al., 2015; Zhou et al., 2015] have largely focused on labeled image and tabular data sets where classes are clearly separable and less so on time-series data in general. Recent work has focused on using prototypes to provide in-process explainability of classification models, either by learning meaningful pixels in the entire image [Li et al., 2017] or by applying attention through the use of sub-regions or patches over an image [Chen et al., 2018]. Class attention maps (CAMs) provide probability maps to highlight areas of images that lead to a certain prediction [Zhou et al., 2015], but do not give examples of prototypical examples of the data or explanations of how the training data relates to the end result. We focus on the former work [Li et al., 2017] for example-based explainability where the generation of prototypes are intended to look like global representations of the training data.
Time-series classification on 1-D data with deep neural networks is a rapidly growing field, with almost 9,000 deep learning models [Fawaz et al., 2018; Pons et al., 2017; Faust et al., 2018; Goodfellow et al., 2018]. One such example leverages global average pooling to produce CAMs to provide explainability for a deep CNN to classify atrial fibrillation in ECG data [Goodfellow et al., 2018]. However, the number of available healthcare datasets, specifically ECG waveforms, is limited [Fawaz et al., 2018]. Within this context, time-series classification on ECG waveforms has been done on a small scale, typically with single beat or shortduration (10 s) arrhythmia classification [Faust et al., 2018; Yildirim et al., 2018]. We adopt the autoencoder-prototype architecture from [Li et al., 2017]. Let X = (xi; yi)in be the training set with xi 2 Rp and class labels yi 2 f1; :::; Kg for each training point i 2 f1; :::; ng. The front-end autoencoder network learns a lower-dimension latent representation of the data with an encoder network, f : Rp ! Rq. The latent space is then projected back to the original dimension using a decoder function, g : Rq ! Rp. The latent representation, f (x) is also passed to a feed-forward prototype network, h : Rq ! RK , for classification. The prototype network learns m prototype vectors, p1; p2; :::; pm 2 Rq using a fourlayer fully-connected network over the latent space that learns a probability distribution over the class labels yi (Fig 2). The learned prototypes can then be decoded using g and examined to infer what the network has learned. The choice of m is determined a priori, with larger values allowing for higher throughput and model capacity.
We improve prior work by adding a penalty for learned prototypes in the objective function of the above network to increase prototype diversity and coverage of the data in latent representations. To align with the minimization of the objective function, this new prototype diversity penalty needs to be (1) small when distances between prototypes are far apart, and (2) large when distances between prototypes are close in distance. We can evaluate the feasibility of a set of prototypes by considering the distance of the two closest prototypes across all prototype combinations. So, we consider the average minimum squared L2 distance between any two prototypes, pi; pj for our loss function. To achieve the desired property above, we take the inverse of this average distance: P DL(p1;:::; pm) = 1 log m1 Pm j=1 mini>j2[1;m] kpi pj k22 + The logarithm function tapers large distances so that the penalty does not quickly vanish, and the term is for numeric stability. By taking the inverse of the log of the prototype distances, we penalize prototypes that are close in distance while making sure the minimum distance between prototypes does not get too large. This prototype diversity loss (PDL) promotes coverage over the latent space. We update the objective function to:
L((f; g; h); X) = E(h f; X) + R R(g f; X) (1) (2) + 1 R1(p1; :::; pm; X) + 2 R2(p1; :::; pm; X) + pd P DL(p1; :::; pm) where E is the classification (cross entropy) loss, R is the reconstruction loss of the autoencoder (i.e. L2 norm), and R1 and R2 are the loss terms that relate the distances of the feature vectors to the prototype vectors in latent space [Li et al., 2017]:
R1(p1; :::; pm; X) = R2(p1; :::; pm; X) = 1 Xm m j=1 n 1 X n i=1 mini2[1;n] kpj
f (xi)k22 ; (3) minj2[1;m] kf (xi)
2 pj k2 (4) The minimization of the R1 loss term promotes each prototype vector to learn one of the encoded training examples, while the minimization of R2 loss promotes encoded training examples to be close to one of the prototypes. This balance gives meaningful pixel-to-pixel representations between the prototypes and training data.
We train our models with a randomly shuffled batch size of 100 (ECG, Speech) and 125 (Respiration). We parameterize the number of prototypes (see supplement) and the regularization term pd for the classification tasks while keeping the other hyperparameters as in [Li et al., 2017].
We adopt a version of the group fairness metric presented in [Mehrotra et al., 2018] and refer to it as the prototype diversity score, : t = 1 X p
Z i=1 j ij (5) where i; i 2 f1; :::; tg is defined for a specific metric and Z is the normalization constant. For the neighbor diversity metric N , i is the set of prototypes that have nearest neighbor i and Z is the number of prototypes m. For the class diversity metric C , i is the set of prototypes that are from class i and Z is the number of classes K. Higher scores will occur when prototypes have more unique elements. Thus, max( D) = 1. 2.3
The neonatal intensive care unit (NICU) dataset is composed of two sources: (1) ECG and Respiration waveforms from PhysioNet’s PICS database [Gee et al., 2017; Goldberger et al., 2000]; and (2) ECG waveforms (500 Hz, Intellivue MP450) collected from a preterm infant over their entire stay ( 10 weeks) at Seton Medical Center Austin. The inclusion of (2) helps supplement the ECG events from (1). The image data used in this study are made publicly available1.
The inter-breath intervals (IBIs) from the respiration were extracted using a standard peak finder. The respiration signals were clipped into 60 second segments that were normalized to zero-mean, unit variance. The R-R intervals for the ECG of the NICU dataset were extracted using a Morlet wavelet transformation of the ECG signal. An open-source peak finder was applied to the wavelet scale range (0.01 to .04 scales) related to QRS complex formation in the spectrogram. The ECG waveforms were clipped at 15 seconds with the event in the middle. All ECG segments were bandpassed filtered from 3 to 45 GHz, scaled to zero-mean, unitvariance, and scaled to the median QRS complex amplitude. Images were then captured to mimic what a clinician would see upon investigation of an ECG signal. Waveforms with no
visibly distinguishable QRS complexes or respiratory peaks were discarded because these waveforms are too obscure for even a clinician expert to evaluate.
Class breakdowns for bradycardia in the ECG signal follow clinical thresholds [Perlman and Volpe, 1985]: XECG = f normal (>100 beats per minute (bpm)): 1039, mild (100-80 bpm): 634, moderate (80-60 bpm): 306, severe (<60 bpm): 132 g. Moderate and severe events were combined into a single class. The class breakdown for apneas in respiration are: XRESP = f normal (1-3 s): 1939, mild (4-6 s): 1921, moderate/severe (> 6 s): 1487 g.
The Free Spoken Digit Dataset [Jackson et al., 2018] consists of 2000 audio clips (8 kHz) of four speakers repeating the digits 0 through 9, 50 times each. Each segment was normalized to zero-mean, unit-variance and clipped for white space (Fig. 3). This data can be thought of as “spoken MNIST”. We perform speaker classification and digit classification within a speaker. 2.4
We use PCA to reduce the latent space vectors to a dimension of 500, which retains 98% of the variability. We then calculate the cosine similarity between these 500 dimensional vectors to produce a similarity matrix and use t-distributed stochastic neighbor embedding (t-SNE) from [Van der Maaten and Hinton, 2008] to reduce the 500 x 500 similarity matrix down to three dimensions for visualization purposes. This technique calculates the KL-Divergence between the higher-order dimensional latent space and the lower dimensional space used to represent the former visually. This approach is non-deterministic so the global position in the lower space is uninformative and instead proximity to neighbors is the key insight to gain. Additionally while the first two dimensions of the projection show the general spread of information, the second and third dimensions maybe useful for visualizing within group information. Thus, we use the second and third dimensions for our visualizations. 3 3.1
We test our prototype implementation on ECG waveforms related to bradycardia using the NICU data for a 3-class classification task using 10 prototypes. We treat the input waveforms as 2-D images and use a four-layer autoencoder to learn complex representations over the data.
We observe more diverse prototypes and comparable or better test accuracy with our model 93.1 0.4% compared with 92.1 0.1% from the baseline model in [Li et al., 2017] (Table 1). Both models perform well on the classification of the normal class, as expected since normal waveforms have near-constant phase. Both models additionally have difficulty separating between the mild and moderate/severe classes, often confusing the classification between these two (see supplement). This behavior is expected since data near these ECG: Bradycardia pd 0 500 1e3 2e3
Accu. 92.1 92.7 92.4 93.1 0.1% two class boundaries are difficult to discern, even for domain experts, due to events existing in both classes with possible subtle time differences in cardiac firing. Our model also improves prototype diversity (Table 1) over the baseline model. This result suggests that the prototype diversity loss encourages exploration, through learning diverse prototypes, within the data represented in the latent space. As a result, our model finds more helpful features and prototypes and thus, improves classification results.
Because prototypes are generated during training, we infer features that the algorithm utilized to classify waveforms at different points during training (Fig 5). For example, by epoch 100, we see that some of the prototypes exhibit global morphological features of the normal waveform class after random initialization at epoch 0. As training progresses, we observe other complex phenotypes emerging: one prototype learns that large gaps in cardiac firings are important for identifying severe cases and another prototype learns the consistent pattern of spikes are important for mild cases. Since the mild class shares mixed features of both normal and positive events, it is not surprising that more prototypes are needed in this class to learn subtleties of the class features (see supplement). Thus, prototypes highlight waveform structures that the algorithm deemed as important when trying to learn the classification of bradycardia. This finding aligns with the idea of clinicians using visible features present in a bradycardia (i.e. the increasing distance between QRS complexes) to decide whether or not a bradycardia exists in an image.
We compare the latent space of [Li et al., 2017] to the latent space of our model with prototype diversity loss via tSNE projections, where proximity in 2-D space suggests that points are “close” in distance in the original latent space. We represent the learned prototypes by mapping each prototype to its nearest neighbor (Fig 4). We find that by increasing our loss term, P DL, our model increases the local coverage of the prototypes compared with the baseline model (i.e.
pd = 0). However, if we regularize our loss term too much (i.e. pd > 104), we begin to introduce clustering of prototypes and diversity suffers. Thus with the additional prototype distance penalty, we achieve higher diversity scores and classification accuracies for various hyperparameters (Fig 9). 3.2
We observe that ECG events in a local neighborhood share similar QRS complex morphology, despite having different class labels and cardiac firing periods (Fig. 6, bottom). Even though we did not impose a class constraint, we observe that the algorithm found two separate features within the moderate/severe class that were important in the classification task (i.e. prototypes 2 and 10 shown at the top of the (Fig 6). These two prototypes explore two different cardiac timings as prototype 2 exhibits a progressive delay in cardiac firing, while prototype 10 exhibits a large spontaneous delay. The incorporation of the prototype diversity loss encouraged this exploration of the latent space. These results suggest that there are physiologic dependencies (i.e. clustering based on cardiac morphology and function) that can be learned using our model to investigate physiological phenomena, and possibly applied to other clinical areas, like cardiac ischemia or apnea of prematurity in respiration - both exhibit visible, abnormal waveform behavior. This work provides a visualization tool for clinician experts to evaluate different morphological of physiological time-series data2. 3.3
Apnea of prematurity is common among preterm infants, and is visually apparent as a pause of inhalation and exhalation (i.e. absence of sinusoidal behavior) in the respiratory signal. We next test our prototype implementation on respiration
waveforms that are related to apnea in a 3-class classification task. We treat the input waveforms as 2-D images again, since clinicians evaluate apneas through visual inspection of the respiration signal.
We observe more diverse prototypes and comparable or better test accuracy with our model 82.3 3.8% compared with 81.4 3.6% from the baseline model, and with overall unique nearest neighbors ( N = 1) and class diversity ( C = 1) (Table 1). Both models have difficulty separating between the event classes because data near these two class boundaries are difficult to visually discern (i.e. 6 second gap versus 7 second gap) and have common behavior with regular respiratory function that is found in the normal class. We find that the addition of a prototype diversity loss maintains or improves performance and yields more diverse prototypes (Table 1).
We also note that the algorithm is able to discern physiological examples and generate learned prototypes that distinctly relate to physiological behavior. For example, in Fig. 7, we see that algorithm finds segments that are related to periodic breathing of 9 second duration (moderate/severe). These segments are physiologically different from normal apneas of 6 seconds (mild), and clearly different from normal breathing with periodicity of 1 second (Fig 7). In the set of eight learned prototypes, the algorithm finds three different classes easily, each with different respiratory phenomena, that are critical in the classifying various types of apneas. 3.4
Speech abnormalities can be suggestive of underlying pathological dysfunction, and common features that clinicians visibly discern in waveforms to assess speech include cadence, prosody, and syllable articulation. To aid in speech feature detection, we assess our model on high-frequency audio waveforms of spoken digits (FSDD) from medically-normal individuals. These digits are treated as 2-D images for 4 class speaker and 10 digit classification tasks with 4 and 10 prototypes, respectively. The waveform envelope and syllables of these spoken digits are discernible to the eye (see “six” and “se-ven” in Fig 2) and, as such, make good candidates for our image-based explainability model. We demonstrate some of the learned prototypes in Fig. 8, which show representations the model finds useful in classifying digits for a given speaker. Experiments show that by varying regularization of the prototype diversity penalty, we observe slightly better or similar accuracies when compared to the baseline model (Fig. 9). With a fine-tuned pd we can increase diversity of the prototypes and correspondingly see improved accuracy and data coverage (see supplement). For example, pd = 500 gives a higher diversity score across all tasks, indicating prototypes with more unique nearest neighbors as compared with the baseline model (Fig 9).
Experiments show that increasing the depth of the network and fine-tuning the learning rate lead to both increased accuracy and diversity over all tasks. Similarly, recent data augmentation techniques in medical [Bahadori and Lipton, 2019] and speech recognition [Park et al., 2019] domains could help further improve performance. The purpose of this work, however, is not to obtain the best performance on these tasks, but rather to show the utility of learned prototypes as faithful explanations of decisions made by a model. 4
We presented a new autoencoder-prototype model that promotes diversity in learned prototypes by penalizing prototypes that are too close in squared L2 distance in the latent space. The new term, pd P DL(p1; :::; pm), in the loss function (Eq. 2) promotes prototype diversity while improving classification accuracy and prototype coverage of data represented in the latent space. These prototypes help explain which global features and representative segments in the training data are most useful for deep time-series classification. This in-process generation of prototypes offers explainable insights into deep classifiers.
Our model and results provide an important significance that previous works lack. Depending on the clinical context of the case, experts may want to either trivialize big differences in the time series features, or conversely accentuate nuanced differences in learned prototypes as clinically important signs of impending adverse outcomes. Therefore, our implementation offers a collaborative method for clinician experts to use their insight interactively with machine learning algorithms: increasing pd promotes large observable differences in the prototypes, while decreasing pd promotes diverse features and prototypes. In turn, our model enables a closed-loop feedback framework to accelerate phenotype discovery to lead clinicians to better-informed decision.
We evaluate the performance of our model on increasingly difficult physiological datasets to demonstrate the effect of pd. The ECG signal is more robust against movement artifact and produces a cleaner signal for the 2-D visualization task, whereas the respiration signal, which is the resultant voltage change across diaphragm movement, is highly susceptible to signal artifact. Additionally, speech waveforms are compressed, high-frequency waveforms (kHz) which make it difficult to visibly extract high-resolution features. We find that our model allocates more prototypes to learn the intricacies of the more indistinguishable classes (i.e. mild and moderate/severe) that are hard for a human to discern, especially the mild cases because this class is a mixture and intermediary of the two extreme classes.
We observe, however, that the high number of loss terms creates a trade-off between prototype interpretability and model accuracy. For example, we observe that for a small number of prototypes, we achieve near-perfect prototype reconstruction but at the cost of classification accuracy. When the number of prototypes was large, we achieve higher accuracy but received noisy prototypes. In future implementations, we can replace the front-end autoencoder with a model that operates well on 1-D time series, like an recurrent neural network, to balance accuracy and prototype interpretability.
There has also been work on computing prototypical patches over 2-D images to generate explainable sub-features [Chen et al., 2018]. Extending the idea of patches to 1-D time-series signals would allow for parsing the signal for subfrequencies and features that could better explain how events are triggered. Nonetheless, the work presented in this paper provides a more robust prototype model to help explain algorithmic behavior and decision-making in deep time-series classification tasks with promising results in clinically relevant datasets.
The authors would like to thank Sinead Williamson and the three reviewers for providing helpful feedback and critical reviews of our work.