Imputing missing values in imbalanced datasets remains an open challenge. Most methods assume data are missing at random or follow a standard distribution, lacking robustness for complex real-world data. Electronic health records exhibit severe class imbalance with non-random missingness, hindering model performance. We propose 3-BRITS for greater scalability and flexibility, modeling temporal and cross-feature correlations to impute missing data, by optimizing sample similarity with deep metric learning for self-supervised learning. Evaluating imputation alone avoids reduced diversity and model bias from joint downstream tasks. Our model achieves superior performance to all baseline methods on four real-world datasets. This shows promise for increasing model scalability and flexibility to handle complex real-world data.
The development of successful machine learning models,
leveraging data procured from Electronic Health Records
(EHRs), necessitates confronting the prevalent issue of
data missingness inherent in this extensive information
repository. Nevertheless, the assorted data types
encapsulated within EHRs are gathered at variable time
intervals, mirroring both clinical and administrative decisions
enacted by healthcare practitioners to bolster patient
care. Consequently, the data amalgamates both static
and irregularly-sampled temporal data. For instance,
vital signs like heart rate are often subject to regular
monitoring, while the recording of white blood cell (WBC)
count is not universally consistent across patients. This
is primarily because it is infrequently ordered for
clinically stable patients who are minimally suspected of
harbouring an infection [
The complex phenomenon of missingness in EHRs poses significant challenges to imputation algorithms.
Beyond the high rates of missing data, over 50% of EHR
information is missing non-randomly [
welldocumented in retrospective studies
evaluating long-term missingness patterns in multi-center medical
In these analyses, significant variation was
observed across tasks, variables, and time [
Deep learning (DL) models, encompassing convo- LIM [
Our study aims to address these gaps in managing such biases and enhance data restoration eficiency.
missing data in Electronic Health Records (EHRs) by
extending the RNN model BRITS [
Our model, 3-BRITS, employs BRITS as a backbone numerical ( ), categorical ( ), static ( ), and
dyand adaptively learns properties intrinsic to clinical time- namic ( ) variables. It is crucial to note that ℝ is
series data through self-supervised deep metric learning. heterogeneous, encompassing structured data types that
3-BRITS surpasses all existing imputation methods, extend beyond purely numerical features. This
configuincluding BRITS, in realistic complex scenarios. ration allows for a comprehensive representation of the
diverse data elements inherent in complex multivariate
time series.
2. Related Work Information related to missing values is encapsulated
within two derived matrices (see Fig. 1). The mask
matrix ∈ ℝ × indicates whether each element of is
observed or missing:
Eforts to impute multivariate time series data have
resulted in numerous strategies. Among these, the GRUD
model [
Existing temporal imputation methods struggle in tures in the data , there is a corresponding variability
managing class imbalances and disparate missingness in . The ∈ ℝ × encodes the time gap between two
distributions, often evident in clinical data [
The definition of this indicator follows: = − −1 ⎨ ⎩0 ⎧ − −1 + −1 if > 1, = 0 if > 1, = 1 if = 1
Our work extends BRITS’s assumptions, emphasizing temporal and feature correlations embodied in . BRITS architecture, combining a fully-connected regression module and a recurrent component, applies temporal decay (Eq. (3)) and a decay factor to handle temporal correlations and adjust influence based on temporal distance.
Missing values within an observation are managed via a historical representation ̂ and a masking vector , producing a complement vector ℎ that accounts for missingness patterns (Eq. (3)-(6)).
ℎ = exp (− max(0, ℎ + ℎ )) ℎ̂ −1 = ℎ−1 ⊙ ℎ
̂ = ℎ̂ −1 + ℎ = ⊙ + (1 − ) ⊙ ̂ (3) (4) (5) (6)
BRITS explores intra-observation correlations through a fully-connected layer, generating approximation of missing values (Eq. (7)). The concept of decay extends to feature space, resulting in a learnable factor, ̂ , considering both temporal decay and the masking vector (Eq. (8)-(9)). This integration produces the imputed matrix , efectively combining observed
, a feature-wise and imputed data (Eq. (10)-(11)).
= ℎ + = exp (− max(0, + )) ̂ = ( [ ∘ ] + ) = ⊙
+ (1 − ) ⊙ ℎ
= ⊙ + (1 − ) ⊙
(7)
(9)
(10)
underexplored [
The final step (Eq. (12)) updates the hidden state via Recurrent Neural Networks (RNNs), leveraging various indicators to learn functions of past observations. The bidirectional recurrent dynamics approach integrates backward information to tackle slow convergence, providing paired outputs , ; and ℎ , ; ℎ (Eq. (13)).
In essence, BRITS exploits temporal and feature
correlations in multivariate time series data, employing decay
factors, a regression module, and a bidirectional RNN for
imputing missing values. The final hidden states are
updated using imputations and corresponding masks, with
(2) the integrated processes visualized in Figure 2.
and flexible imputation through the use of 3-BRITS, as
illustrated in Figure 3. In real-world temporal data, the
degree of heterogeneity varies across diferent domains. For
instance, healthcare data often demonstrate significant
class imbalance. Furthermore, the time-varying statistics
of various types of data present substantial challenges
for interpretation within a single location, a complexity
that amplifies when extended to multiple locations. This
makes the incorporation of all relevant prior knowledge
into the analytical model impractical. Traditional metric
learning techniques [
To navigate this complexity, we extend the base model by applying Self-Supervised Deep Metric Learning to diminish task bias while facilitating the exploration of individual properties. and generating a diverse array of examples, the model can analogous to BERT [31]. We initially prepend a learndiscern valuable features distributed throughout the en- able embedding, the [CLS] token, to the hidden states tire data, inclusive of those from minority classes, thereby as delineated in Equation 14. Each transformer encoder enhancing the overall representational richness. This layer comprises two sublayers: (a) a multi-headed selfstrategy can yield considerable benefits, especially under attention mechanism (MSA, as defined in Equation 15), conditions where labeled data is either scarce or imbal- and (b) a feed-forward network (FFN, as defined in Equaanced. Furthermore, self-supervised learning can provide tion 16). Residual connections [32] are utilized around assistance in alleviating issues associated with overfit- each of the sublayers in both the MSA and the FFN, folting, which are frequently encountered when models lowed by layer normalization (LN). The [CLS] embedding are trained on imbalanced datasets. Notwithstanding serves as , , . these advantages, challenges might emerge, including the identification of suitable auxiliary tasks and the risk ℎ̂ = [CLS; ℎ0; ℎ1; ...; ℎ ] (14) of learning non-discriminative features [30]. ℎ̂ ′ = LN(MSA(ℎ̂ ) + ℎ̂ ) (15) Online Hard Triplets Mining The directionality of representations, whether originating from single-layer , ∈ [ ( ( )), ( ( ))] (17) BRITS or our 3-BRITS model, is employed to guide the ∈ [ ( ( )), ( ))], ≠ construction of triplets. The dual directional indicators double the mini-batch size. This strategy applies to both where the represents the forward hidden states of labeled and unlabeled data, ensuring each triplet is bal- sample , for backward. Among the constructed anced with one positive and one negative sample pair. triplets, we select all triplets that violate the following However, this paper primarily focuses on labeled data in condition: an efort to address the imbalance problem. Specifically, ∥ − ∥2 + < ∥ − ∥2 (18) one representation is assigned as the anchor (denoted as ), and another randomly chosen representation from is a pre-set margin, meaning we only consider samples the same group is defined as the positive sample (denoted that are easily confused with the anchor sample by a as ). Concurrently, representations from a diferent margin . group within the same mini-batch are classified as negative samples (denoted as ).
Loss Function This study utilizes the state-of-theart Multi-Similarity loss (MS loss) objective function, a Incorporating Self-Attention for Optimal Represen- renowned metric learning objective [33]. The MS loss tation Learning In order to optimally learn repre- function prioritizes the significance of the samples by sentations in each direction, we employ a methodology exploring the similarities within positive pairs and between negative pairs. The pairs that are most informative ℎ̂ ∗ = LN( ( ℎ̂ ′) + ℎ̂ ′) (16) are often indistinguishable and hence contribute more substantially during the training process.
ℒ = 1 ∣ ∣ ∈ ∑ { 1 log[1 + ∑ ( S −) ]
(19) + 1 log[1 + ∑
( S −) ]} ∈ ∈ where , , are hyperparameters; denotes mini-batch samples.
In this section, we carefully assess and analyze the performance of 3-BRITS by comparing it with state-ofmains of healthcare, environment, and trafic. The models against which 3-BRITS is compared include BRITS, GRUD, V-RIN(full), and MRNN. For all experiments, only the best model from each study is employed for comparison. While a comparison with the 2GAN model would have been intriguing, especially considering that [34] does not provide a quantitative comparison with BRITS, its inclusion in our study was precluded due to the obsolescence of the publicly-available code based on TensorFlow 1.7 and Python 2.7, rendering it incompatible with our accessible GPU hardware.
The four datasets chosen for experimental evaluation each exhibit distinctive data distributions, with the MIMIC-III database particularly noted for its high rate of missing data. In our replication of the benchmarking studies for these publicly accessible datasets, we opted to bypass any steps that involved the removal of all-NaN samples to preserve the original missingness inherent in the data.
In this study, we utilized the Adam optimizer across all models, with the number of RNN hidden units fixed at 108. The batch size was determined based on the specific dataset; MIMIC-Mortality and Air Quality were allocated a batch size of 128, while Trafic and MIMIC-Physionet Challenge were assigned a batch size of 64. To promote stable training, each dataset was normalized to have zero mean and unit variance. Randomly, we selected 10% of each dataset for validation and another 10% for testing, training the models on the remaining data. For the imputation task, we randomly masked 10% of observations in each dataset to serve as the ground truth, which was used as validation data. A 5-fold cross-validation method the-art models using four real-world datasets in the do- (MAE) and Mean Relative Error (MRE). As for the Muland -BRITS. The 3-BRITS model represents Multiple Masking Metric Learning. Similarly, the 2-BRITS denotes Multiple Masking Learning, and the -BRITS variant only encompasses Metric Learning. These diferent versions allowed us to evaluate the efectiveness and contribution of each component in a variety of real-world datasets. outperformed the others across all datasets. However, it is noteworthy that, for each dataset, the MREs of our proposed 3-BRITS and 2-BRITS models were generally equivalent to, or better than, those of the other competing models. Baseline models, namely MRNN, GRUD, and V-RIN-full, did not outperform our proposed models in terms of either MAE or MRE across all datasets. This underlines the eficacy of the
3-BRITS and 2-BRITS methodologies in processing these real-world datasets.
We propose 3-BRITS, an innovative architecture that incorporates deep metric learning into the BRITS model for imputing missing values in multivariate time series characterized by non-random missingness. Our empirical ifndings indicate that 3-BRITS delivers state-of-the-art performance across various datasets in the realm of time series imputation.
The 3-BRITS model advances the BRITS architecture by optimizing sample similarity through self-supervised deep metric learning. This approach allows for circumventing prevalent issues such as class imbalance, thereby averting the imposition of artificial data distribution. By assessing imputation performance in isolation, as opposed to jointly with downstream tasks, 3-BRITS prevents the diminution of diversity and model bias that can emerge from varying data focuses.
The experimental results substantiate that 3-BRITS outperforms all baseline models. These outcomes underscore how augmenting an imputation model with deep metric learning and self-supervised learning empowers it to manage more intricate data with non-random missingness, an area where other techniques might falter. The 3-BRITS model presents a promising prospect for achieving superior scalability and flexibility to address challenges encountered in real-world time series data.
In forthcoming research, we aspire to evaluate the performance of 3-BRITS on additional complex healthcare datasets, particularly those typified by high missing rates, class imbalance, and mixed variable types. We will also explore potential enhancements to 3-BRITS, inclusive of the incorporation of other self-supervised learning methodologies.
This paper represents independent research funded by the NIHR Maudsley Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The work of Linglong Qian was supported by the Kings-China Scholarship Council PhD Scholarship Programme (K-CSC) under Grant CSC202008060096. All models are implemented on CREATE HPC.1 1King’s College London. (2022). King’s Computational Research, Engineering and Technology Environment (CREATE). Retrieved March 2, 2022, from https://doi.org/10.18742/rnvf-m076